PerfPilot

vLLM ships performance regressions silently. One commit can cost 10–60% of a model's throughput, meaning you pay for up to 2× the GPUs. A release has hundreds of commits, every test is a GPU run, and noisy runs lie, so engineers blame the wrong commit. PerfPilot is an agent that finds the culprit commit and proves it. Give it a good and a bad vLLM commit. It bisects the range on a real A100 (Modal) serving Meta's Llama-3.2-1B and returns the culprit, a statistical interval (log-ratio Welch) and the GPU dollars spent. TuneSmith then tries serve flags to win the speed back, and accepts a flag only when the gain is proven above +2% and a quality gate holds. The agentic loop is a proposer swarm on Meta's open-weights muse-spark-open: a bisector, a code reader that reads real commit messages, and a statistician vote on which commit to probe next. A deterministic referee picks, the GPU measures, statistics decide. A bad proposal can waste one measurement but never change a verdict. Moat: proof, not guesses. A 17-scenario gym (noise, collapsed runs, broken builds, flaky workers, rig drift, misleading commits) scores every change against git SHA, and it caught real bugs we fixed, including GPU workers dying because the cache volume ran out of inodes, not bytes. Results: 0 wrong verdicts across 340 scripted runs, the live-model bench and every GPU run. A real A100 bisect: 16 commits narrowed to 2, −61% effect, $1.23. 228 tests. Reports re-render offline.