# Lucky Passes

- **Event:** [Meta Global AI Developer Hackathon](https://cerebralvalley.ai/e/meta-ai-developer-hackathon)
- **When:** Oct 26, 12:00 PM – Nov 4, 9:00 PM (EST)
- **Where:** Online
- **Team:** [Shailendra Bade](https://cerebralvalley.ai/u/finaspirant)
- **GitHub:** https://github.com/finaspirant/lucky-passes
- **Demo video:** https://www.youtube.com/watch?v=kCBHWg1bPgI
- **Gallery:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1410

When a coding agent’s attempt passes every visible test but fails the hidden ones - a “lucky pass,” can an  LLM judge (Meta Muse Wonder) catch it, and does reading the agent’s process help? 
I built an evaluation harness from scratch: Muse Spark implements 16 functions from written specs (10 attempts each), hidden tests decide correctness, and Muse Wonder judges each attempt two ways -  from the full agent transcript, or from the final code against the spec alone. 

I preregistered the hypothesis that process-reading would win. It didn’t: code-against-spec review scored 0.93 AUROC vs 0.51 for process (directional; lucky passes came from 2 of 16 tasks). On one task all 10 attempts shared the same blind spot, so best-of-N sampling couldn’t help, but the code judge named the EXACT violated requirement every time, and it found REAL bugs our own hidden tests missed. 
Lesson for anyone shipping coding agents: retries don’t catch shared blind spots but a spec-grounded review  by powerful LLM model like Muse Wonder can. (Includes a public interactive viewer of every attempt and judgment.)

## More from Meta Global AI Developer Hackathon

- [Tigers](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1407)
- [Pay Agentic Ai-Dhanish](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1408)
- [Daniel Rojas Vidal - RoyaLens](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1409)
- [Muse Live](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1411)
- [Project Checkpoint](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1412)
- [Delta-V](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1413)

---

Markdown version of https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1410. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt