Nile

A model scores 95% accuracy. Can you trust it? Often not: leakage between training and test data quietly inflates results, and it is common even in published research. ML Experiment Auditor is an AI agent that checks how a machine learning experiment was evaluated, finds what inflates the score, fixes the evaluation, and measures the honest result, with evidence for every number. The idea came from my own Formula 1 pit-stop model: 95% accuracy on a random split, 62% when tested on a new season. That drop alone proves nothing, because the label mix changed too. I needed to know what was actually wrong, and by how much. How it works: "the model reasons, code measures." Meta's Muse Spark model (muse-spark-1.3, via the Meta Model API) leads the investigation by calling tools. It reads the code, forms hypotheses, chooses checks, re-runs the experiment with corrected splits, and writes a fix. It never computes a number itself. Every number comes from deterministic code that actually ran: - The project is re-run in an isolated sandbox with no API keys. - A tracer records which rows every fit and predict call saw. - Checks for six kinds of problems: preprocessing fitted on test rows, features not known at prediction time, the same entities or later dates in train and test, duplicated rows, choices made on the test set, and a code review of every script. - Muse proposes a fix as a patch. Code rejects any change to the model itself, re-runs it, and re-checks. - Guardrails: every claim must cite evidence, and if Muse writes a number no tool produced, the verdict is rejected and sent back. "Inconclusive" is a valid answer; it does not accuse without proof. The output is a report: verdict, findings ranked by severity with the exact code lines, the fix as a diff, the corrected score next to the original, how far the score can be trusted, and recommendations. Results: on my F1 project it found that all 48 test races were also in training and that 321 test laps duplicated training laps, then fixed the split. Honest accuracy: 0.705, not 0.952. On a 20-project benchmark with planted flaws and clean controls (6 held out, answers locked before any run): 20/20 correct, zero false alarms, and 13/13 fixes landed on the true score (mean error 0.0009). Muse alone, without tools, is off by about 5 points when it guesses the real score. A typical audit costs about $0.05 to $0.15. Built during the hackathon in Python (scikit-learn, pandas), with a command line and a local web page where you upload a project and watch the investigation live.