Upgrade Gate

Upgrade Gate: before you switch to a new model version, check how it fails, not just how often it's right. The problem: small shops in India take orders as voice notes in Hindi, Hinglish and English. If an app like that upgrades its AI model and the new model starts saving the wrong pincode, nobody notices until parcels go to the wrong city. Most teams decide on an upgrade by checking a pass rate. We found a case where the pass rate doesn't move but the failures become invisible. What it does: Upgrade Gate replays your own tasks on the old and new model, repeats every call, and turns the differences into GO, NO-GO or INCONCLUSIVE, with the failing answers attached. It uses paired exact tests, confidence intervals and corrections across language, input type and task type, and it checks HOW answers fail: a model that says "I can't hear this" is safer than one that returns a correctly shaped wrong answer. It runs as a GitHub Action, so the pull request that bumps the model fails its check and the deploy step never runs. We built VoiceOrder, a small voice-order app, as the thing it protects. What we found (real Meta Model API runs, $26.93 in all): - Voice notes, Spark 1.2 to 1.3 (135 synthetic clips): accuracy barely moved (278 vs 274 of 405 answers), but wrong or blank answers with no warning rose from 18 to 42 (p = 3e-6). 1.2 says "I can't hear the audio"; 1.3 never does. An accuracy-only gate says INCONCLUSIVE; Upgrade Gate says NO-GO. - Real voices (two speakers, 90 clips, predictions written down before each run): accuracy fell 8.9 points, so NO-GO. The silent-failure rise did not repeat for the second speaker, and we say so. - A safer route: Muse Voice Transcribe first, then Spark 1.3, passes on synthetic notes (+12.8 points) and on the new speaker (+4.4). Pooled real voices are still inconclusive, so that upgrade stays blocked too. - More reasoning effort made 1.3 worse at listening: 72.8% right at minimal vs 58.5% at xhigh, at 6x the cost. - Contributor mode: same pass/fail as Standard on all 1,325 text tasks, at $0.10 vs $1.80 per 1,000. - We caught our own scorer inventing a "significant" Hindi gain for 1.3. It was spelling (Somwar vs Somvar). Try it: live site https://web-production-00176.up.railway.app (VoiceOrder at #/voiceorder), the blocked upgrade https://github.com/UtsavInTech/upgrade-gate/pull/1, every result with caveats in findings/LOG.md. Limits: single-turn tasks only; real-voice evidence is two speakers and 15 scripted items; no outside team has run it on their own workload yet. Built during the event from 27 Oct 2026. I designed the study and ran the experiments; Claude Code and Codex helped write the code.