crosscheck

Crosscheck checks research papers like a tireless reviewer: it recomputes every reported statistic, then reruns the analysis every reasonable way to see whether the conclusion holds. The need is enormous. Nearly nine million researchers publish 3.3 million papers a year, and peer review takes over a hundred million hours a year, yet almost nobody reruns the numbers. In psychology, half of papers misreport a p-value; one in eight, badly enough to change the conclusion. When 29 expert teams analysed the same dataset, 20 found an effect and 9 didn’t. Crosscheck serves PhD students before they submit, reviewers before they decide, and journals screening submissions. It runs in a browser, reads CSV, Excel, Stata, SPSS and R data, and with open weights a lab can self-host it for unpublished work. It already works on real science. On a peer-reviewed PLOS ONE experiment with 985 participants, Crosscheck read all 14 pages, recomputed 24 statistics and found one that doesn’t add up: the paper prints p = .010, but its own F statistic gives p = .099, matching the authors’ own word, “unaffected”. Likely a typo, but it flips significance, and it passed peer review. With the paper’s public data, Crosscheck rebuilt the method from quotes on the page, reproduced the headline test (p = .003 vs the paper’s .004) and ran 96 locked alternative analyses: all agree, the main finding is robust. Upload to verdict: under five minutes. We found this paper by running Crosscheck’s p-value check across 800 PLOS ONE papers. Every agent runs on one open-weight model, Muse Spark Open, served on Modal, in seven roles: reader, spec writer, planner, critic, coder, repairer and explainer. The rule is strict: the model reads and plans, code computes every number. The reader pulls statistics from text, tables and even screenshots of software output. The spec writer must quote the page for every detail or mark it “not stated”. The critic rejects invalid alternatives, like clustering on two groups or controlling for a mediator. The coder writes its own independent version of the analysis, which must agree with ours. SAM 3.1, a segmentation model built for photos, finds the dots in a scatter plot, so Crosscheck recomputes the correlation from the chart itself: 0.94, where a planted caption claims .49. Asked as a plain chatbot, with the data, for a nest-level p-value, the same model said it couldn’t know without running code; Crosscheck runs that code. Every version runs as real code in an isolated sandbox with an import allowlist and a time limit. The plan is hashed and locked before anything runs, so no one, not even the AI, can steer the answer. Checks allow for rounding, so only genuine inconsistencies are flagged, always as “possible”, never “wrong”. The scorecard: 10 of 10 planted errors caught, zero false alarms on clean and real papers, identical numbers across full reruns, and 38 automated tests. Recorded model replies let anyone reproduce the demo without API keys.