# Confam

- **Event:** [Meta Global AI Developer Hackathon](https://cerebralvalley.ai/e/meta-ai-developer-hackathon)
- **When:** Oct 26, 12:00 PM – Nov 4, 9:00 PM (EST)
- **Where:** Online
- **Team:** [Ololade Ehuwa](https://cerebralvalley.ai/u/Ololadestephen)
- **GitHub:** https://github.com/Ololadestephen/Confam
- **Demo video:** https://www.youtube.com/watch?v=patd96UOovc
- **Gallery:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1570

Coding agents now write fixes and report success through tests. But a test that only checks for a "Booking confirmed" message will pass a fix that saves nothing. I hit this building my own app: a report said three out of three tests passed, while required refreshes and intermediate checks were never performed.

Confam challenges AI-generated fixes before you ship them. It asks the questions after "did the tests pass?": would those tests have caught a fix that only looks right, and what did everyone forget to check?

Recorded evidence: https://useconfam.vercel.app/

1. A bug report, or a screen recording, becomes a short list of expectations the developer confirms.
2. Confam reproduces the bug on the real app. It must fail for the reported reason, not a timeout or a missing button.
3. It plants decoys: plausible wrong fixes written by Muse Spark. An independent probe, such as reading the database directly, must confirm each decoy is really wrong, or it isn't counted.
4. It strengthens the checks only where a decoy got through, at most twice. Any change to what "fixed" means goes back to the developer.
5. It freezes the checks by hash, judges the candidate fix (yours, Muse Wonder's or any agent's), and reports raw counts with their limits.
6. A gap finder asks Muse Spark what the expectations, or a proposed fix, still miss. Nothing counts until the developer confirms it and Confam reproduces it.

Results on three real applications. Linkding, which I'd never touched, was onboarded with configuration only; Mosaic needed one general engine option, a longer workflow timeout. The reproductions are real bugs from each app's history; the gaps are expectations I confirmed that fail on the current code.

- Bluemax, a real solar installer's website in Nigeria, mostly written by an AI app builder (local Supabase copy only, never the live site). Its quote-form spam limit reset whenever the function restarted. Confam reproduced it; the real fix passed; Spark wrote three wrong fixes and the checks caught all 3, each confirmed in the database. The gap finder then found three confirmed gaps in the current code, all failing (spoofed headers get around the limit, typos use up the limit, an unavailable limit lets spam through). Muse Wonder wrote a fix for them, and before anyone accepted it, Spark reviewed that fix and found two more confirmed gaps that fail: ten simultaneous requests were all accepted against a limit of five, and a failed save still used up the allowance.
- Mosaic, my own Next.js and PostgreSQL app: two real bugs reproduced and fixed by Muse Wonder (for one, it wrote the same one-line guard as my real fix). The gap finder found two confirmed gaps in the current sign-in lockout, both failing: sign-up attempts with someone's email lock them out without a single wrong password, and a correct sign-in doesn't forgive earlier typos.
- Linkding, a popular open-source Django bookmark manager I have no connection to: its real server-side request forgery bug (checking a bookmark made the server fetch internal addresses) reproduced, with 1 internal request before the fix and 0 after Linkding's own AI co-authored fix. The gap finder's one confirmed gap fails (an environment proxy gets around the protection, which Linkding already documents), and one Spark claim about IPv6 addresses was tested and shown false.

The controlled comparison: real apps have one real fix, so wrong rejections of correct fixes were measured on a purpose-built salon app with two correct fixes. On 6 known wrong fixes used during development, the app's existing test caught 0; AI-written tests used as generated caught 2 but rejected both correct fixes; Confam caught 5 with no wrong rejections. On 4 unseen wrong fixes written after the checks were sealed, by a separate AI agent never shown the checks, and evaluated once with the sealed engine: existing test 0, raw AI tests 2 (again rejecting both correct fixes), Confam 4 of 4 with no wrong rejections. Caveats: those 4 came from the same model family, not from a person, and some resemble the development fault types.

A repair loop puts the two models against each other: Muse Wonder fixes every confirmed gap at once, Confam runs all the cases (including guards for what already worked), Wonder gets one more try told only what it missed, and Muse Spark then reviews the passing fix. On Mosaic and Linkding, Wonder's repairs passed every case, and Spark's reviews each found a confirmed gap the repair still fails: Mosaic still locks out a new user because of failed attempts made before they registered, and Linkding still reaches an internal service when the proxy resolves a public name to it. In a second round Wonder closed both. Mosaic's fix was clean and Spark found nothing more worth confirming; Linkding's passed every check by switching proxies off entirely, and Spark's review caught that cost, confirmed by running the app.

Developers can use it in a local web interface (add an app, turn a report into confirmed expectations, run and read the evidence, confirm or reject Spark's proposed gaps, and follow repair sessions), from the command line, or in CI through a reusable GitHub Action that posts a review of each proposed fix and fails the job when the fix fails.

Everything in the repository was built during the hackathon, starting October 28. Mosaic and Bluemax were built earlier and are test subjects only; Linkding is an independent open-source project. None of their source is in the repository.

## More from Meta Global AI Developer Hackathon

- [Project CAD](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1567)
- [Flamers](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1568)
- [Remembery](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1569)
- [ReARC Labs](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1571)
- [TailorApps](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1572)
- [Letlatsa Pheane](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1573)

---

Markdown version of https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1570. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt