# Nile

- **Event:** [Meta Global AI Developer Hackathon](https://cerebralvalley.ai/e/meta-ai-developer-hackathon)
- **When:** Oct 26, 12:00 PM – Nov 4, 9:00 PM (EST)
- **Where:** Online
- **Team:** [Philip Oyedokun](https://cerebralvalley.ai/u/Mr_Freestyler)
- **GitHub:** https://github.com/Mr-Freestyler21/ml-experiment-auditor
- **Demo video:** https://www.youtube.com/watch?v=YswJspLuNSA
- **Gallery:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3608

A model scores 95% accuracy. Can you trust it? Often not: leakage between training and test data quietly inflates results, and it is common even in published research. ML Experiment Auditor is an AI agent that checks how a machine learning experiment was evaluated, finds what inflates the score, fixes the evaluation, and measures the honest result, with evidence for every number.

The idea came from my own Formula 1 pit-stop model: 95% accuracy on a random split, 62% when tested on a new season. That drop alone proves nothing, because the label mix changed too. I needed to know what was actually wrong, and by how much.

How it works: "the model reasons, code measures." Meta's Muse Spark model (muse-spark-1.3, via the Meta Model API) leads the investigation by calling tools. It reads the code, forms hypotheses, chooses checks, re-runs the experiment with corrected splits, and writes a fix. It never computes a number itself. Every number comes from deterministic code that actually ran:
- The project is re-run in an isolated sandbox with no API keys.
- A tracer records which rows every fit and predict call saw.
- Checks for six kinds of problems: preprocessing fitted on test rows, features not known at prediction time, the same entities or later dates in train and test, duplicated rows, choices made on the test set, and a code review of every script.
- Muse proposes a fix as a patch. Code rejects any change to the model itself, re-runs it, and re-checks.
- Guardrails: every claim must cite evidence, and if Muse writes a number no tool produced, the verdict is rejected and sent back. "Inconclusive" is a valid answer; it does not accuse without proof.

The output is a report: verdict, findings ranked by severity with the exact code lines, the fix as a diff, the corrected score next to the original, how far the score can be trusted, and recommendations.

Results: on my F1 project it found that all 48 test races were also in training and that 321 test laps duplicated training laps, then fixed the split. Honest accuracy: 0.705, not 0.952. On a 20-project benchmark with planted flaws and clean controls (6 held out, answers locked before any run): 20/20 correct, zero false alarms, and 13/13 fixes landed on the true score (mean error 0.0009). Muse alone, without tools, is off by about 5 points when it guesses the real score. A typical audit costs about $0.05 to $0.15.

Built during the hackathon in Python (scikit-learn, pandas), with a command line and a local web page where you upload a project and watch the investigation live.

## More from Meta Global AI Developer Hackathon

- [AudioAID](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3605)
- [Om prakash prajapat](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3606)
- [Andrew Brown](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3607)
- [Call360 Colombia](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3609)
- [Mango](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3610)
- [Usher By Looma](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3611)

---

Markdown version of https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3608. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt