Pit Crew
Gemini 4 NYC HackathonPit Crew is an autonomous agent that makes a repository's CI faster, then proves the faster CI still catches every bug it caught before. The problem: slow CI wastes hours every day, and AI coding agents make it worse by opening more PRs. Asking an AI to "make CI faster" is dangerous. In our test, a single Gemini call deleted real tests or platforms in 6 of 6 rewrites. Faster CI that catches fewer bugs is worse than slow CI. How it works: the agent proposes; deterministic code judges. - Analysis profiles months of GitHub Actions history and finds the slowest step on the critical path. - Optimizer makes ONE change from a playbook (cache, parallelize, shard, dedupe, affected-tests). A static "never check less" guard rejects any change that drops a test, OS, version, trigger or failure recording, or hides failures (|| true, retries, continue-on-error). - Proof picks real past bugs and planted mutations the new CI must still catch. It records the failures on video and in screenshots, and Gemini compares baseline and candidate visually. - Dispatcher + Rater (plain code) run the old and new CI interleaved 5+ times on a fork, replay every bug, and keep a change only if it is statistically faster (p ≤ 0.05) AND catches 100% of bugs. Kept changes open a PR with a proof card. Evaluation vs. baselines: 306 labeled edits on 18 popular repos the guard never saw (Django, Pandas, Ruff, Astro…). - Single model call: 0% of weakening edits caught. - Simple diff rules: at most 33% caught, and the strict one wrongly blocked 30% of safe edits. - Pit Crew: 100% caught (226/226), 0% of safe edits blocked (0/80). Where it failed: we document 5 guard holes we found, plus one unsafe sharding recipe of our own, and what we changed. Runner queue time can't be fixed by editing a workflow, so Analysis reports it as uncovered instead of inventing a fix. The Pit Wall dashboard shows each cycle live.





