Lucky Passes

When a coding agent’s attempt passes every visible test but fails the hidden ones - a “lucky pass,” can an LLM judge (Meta Muse Wonder) catch it, and does reading the agent’s process help? I built an evaluation harness from scratch: Muse Spark implements 16 functions from written specs (10 attempts each), hidden tests decide correctness, and Muse Wonder judges each attempt two ways - from the full agent transcript, or from the final code against the spec alone. I preregistered the hypothesis that process-reading would win. It didn’t: code-against-spec review scored 0.93 AUROC vs 0.51 for process (directional; lucky passes came from 2 of 16 tasks). On one task all 10 attempts shared the same blind spot, so best-of-N sampling couldn’t help, but the code judge named the EXACT violated requirement every time, and it found REAL bugs our own hidden tests missed. Lesson for anyone shipping coding agents: retries don’t catch shared blind spots but a spec-grounded review by powerful LLM model like Muse Wonder can. (Includes a public interactive viewer of every attempt and judgment.)