Triton Kernel Harness

Triton Kernel Agent: an AI agent that writes fast GPU kernels, and a harness it can't fool. LLMs can write GPU kernels, but the speedups they report are easy to fake. Published kernel agents have been caught caching outputs, calling back into PyTorch, quietly dropping to lower precision, or timing only part of the work. We built an agent, and a judge built on the assumption that every kernel is trying to cheat. How it works: given a PyTorch op from KernelBench, Muse Wonder (Meta Model API) writes a Triton kernel to replace it. Each attempt runs in a fresh Modal H100 sandbox with no network and no secrets. Inside it, a trusted parent process never imports the candidate. The kernel runs as an unprivileged user, and only raw tensors pass between the two. The parent checks the kernel against an fp64 reference on the visible shape, two input sets and at least five hidden shapes (odd sizes, size one, very large). It times the kernel on its own clock, swapping inputs between batches and re-checking outputs after timing. A static scan blocks PyTorch fallbacks, TF32 and fp16/bf16, introspection and hardcoded data. A runtime check flags any time faster than the H100's memory bandwidth allows. The result goes back to the model as feedback, with up to 10 attempts per task. We attacked our own harness first. A red team of agents wrote 37 attack kernels; the first version fell to 26 of them, so we rebuilt it with process isolation. With the static scan turned off, the runtime checks alone now stop 32 of the 37, and every attack is kept as a regression test. Results: we ran 43 tasks on an H100 in strict fp32 (12 dev, 31 held out), in three model setups: - Wonder only: solved 43/43; held-out geomean 1.28x faster than PyTorch eager. - Spark only: 42/43 and 1.01x, at about a third of the cost ($0.37 vs $0.85 per correct task). - Wonder writing with Spark fixing compile errors: no better than Wonder alone. On single ops the agent is 1.5 to 1.8x faster than eager; on fused GEMM chains in strict fp32, cuBLAS mostly still wins. On softmax and LayerNorm it matches hand-tuned kernels from the Triton tutorials. Highlights: a LayerNorm that spreads 16 long rows across all 132 SMs (27x eager, 1.5x torch.compile), and a GEMM-plus-sum task the model rewrote as a column sum and dot product (23x). The whole batch cost $73 on the API plus about 4 H100-hours, and no runtime cheat check fired. Every attempt, its code and its result are stored in DuckDB and published as a static results page. Built with: Meta Model API (Muse Wonder 1.0, Muse Spark 1.3), Triton, PyTorch, Modal, DuckDB.