Raveenthecoder

A model scores 97%, everyone trusts it, and nobody checks whether the test measured the right thing. NullLab is a research workdesk for ML researchers, engineers and the reviewers who rely on their results: it lets an author question their own evaluation before anyone else has to. Theme: Build for Your World (a problem every ML author faces: validation that quietly reuses the same machines, sessions or records in training and testing). HOW IT WORKS The author brings a question, a CSV, a compatible Python pipeline and notes. NullLab preserves the original files and inspects their structure without executing the upload. The author's answers become source-quoted draft suggestions; the author reviews and confirms the study brief (target, prediction time, population, input availability). Before anything runs, a concrete plan shows the metric, fixed seeds and proposed comparison, and known incompatibilities stop there, before training or model spend. Meta models sit in the loop: Muse Wonder independently reviews the method, Muse Spark proposes a typed intervention, deterministic guards check applicability and preserve the original classifier, and local execution supplies the measurements. Wonder then challenges the interpretation. Everything is exportable: exact inputs, tested controls, predictions, environment and recorded objections. Reopening restores the investigation, and a changed input cannot inherit an earlier measurement. RESULTS SHOWN IN THE VIDEO 1) A saved synthetic recovery case: the same Decision Tree goes from 0.974 to 0.290 accuracy under rig-disjoint validation. The outcome is labelled inconclusive because row intervals do not calibrate uncertainty across rigs, and the original unsupported attempt stays preserved. 2) A separate recorded example on real UCI SECOM semiconductor data (CC BY 4.0) with a resampling flaw we deliberately introduced: average precision goes from 0.983 to 0.139 on three fixed seeds. The evaluated population and prevalence also change, so we present a protocol comparison, not a pure causal amount of leakage, and Wonder disputes the interpretation. HONEST LIMITS NullLab supports binary tabular evaluation today. It has internal synthetic/API and browser acceptance checks only. We make no outside-user validation, adoption or time-saving claim. Hostile hosted Python and general paper evaluation are unsupported; audits run locally. WHAT'S NEXT Test with outside researchers, and extend uncertainty handling for grouped and time-ordered data. BUILT DURING THE HACKATHON The code, experiments, safeguards and workdesk were built during the hackathon (Oct 26 to Nov 4, 2026); the idea predates it. Third-party: UCI SECOM data, scikit-learn, pandas, NumPy, FastAPI, Meta inference.