Who Judges the Judge

I coding agents can make tests pass without fixing the bug, and teams increasingly ask another AI to judge their fixes. I tested whether that judge holds up against a cheater that reads the judge’s reasons and tries again. Muse Spark plays the cheater and writes fixes that pass the visible tests. Muse Spark Open judges each fix, and hidden tests decide who was right. On bugs from three real libraries (cachetools, requests, Jinja2), the original judge caught 28 of 34 cheats. It missed fixes that looked general but weren’t correct. A stricter prompt that makes the judge find an input that breaks the code caught 46 of 46, with 1 false alarm. The judge also writes a test for each catch, and we run it against the library’s real fix. Every raw result is in the repo and can be checked without an API key.