Litmus

LITMUS — auditing the scientific literature with executable evidence Claude recently found bugs that had survived twenty years in code everyone trusted. The scientific literature is no different. For decades each paper has been checked by one or two reviewers with no time to verify every claim, so errors slip through: conclusions that aren't statistically significant, results impossible under basic thermodynamics, figures that contradict the paper's own claims, numbers that don't add up. More than half of scientists can't reproduce each other's published work, and AI-for-Science agents now build on that same literature as ground truth. LITMUS reads the full PDF, including figures, tables, and supplementary material, and breaks it into a graph of checkable claims. It checks them two ways. For anything numeric, it routes the claim to a deterministic verifier that recomputes the result in code and ships a script you can rerun yourself. For the qualitative problems that no arithmetic catches, like overclaims, extrapolation beyond the data, or causal language on an observational design, Claude reasons about what looks wrong and reports it with a stated confidence level. The two kinds of finding stay separate, so a recomputed proof is never dressed up as a judgment call. What it does - Recomputes checkable errors across chemistry, biology, psychology, and economics: p-values that don't match their own statistics, reaction yields above 100%, totals that don't add up, results that violate basic physical limits. Every flag carries a runnable script and the output it should produce. If a finding has no script, it doesn't ship. - Reasons about the softer failures that no arithmetic check would catch, like overclaims, spin, mismatches between a claim's strength and its evidence, and questionable method choices. Each one is labeled with a trust tier instead of presented as proof. - Routes the genuinely subjective questions to humans rather than scoring them: whether a result matters, whether it's novel, whether it's significant. These get surfaced, not graded, and LITMUS never blurs a human judgment into a machine verdict. - Extends its own verifier library. New checks come from contributors or Claude writes them on the fly, and every one passes the same calibration gate before LITMUS trusts it: catch a planted copy of its own error, clear clean inputs without false alarms, run deterministically, and reproduce in a fresh sandbox, all with no human labels. The library covers more of the field's real errors with each paper and each contributor. How we built it We built it with a Claude Opus 4.8 session running parallel workflows and subagents. The final app uses Claude Managed Agents, combining deterministic workflows with multi-agent parallel execution. A Vercel-hosted web app lets people upload PDFs, and a Supabase backend stores audits keyed by content hash so repeat views are instant. The same pipeline runs as an MCP server, so AI agents can verify papers programmatically. We built it from scratch to a working system in about five hours. Validation On a benchmark of 31 papers spanning psychology, nutrition, chemistry, biology, medicine, and economics, LITMUS confirmed 18 errors by deterministic recompute, including several well documented errors, reproducibly. For example, the Wansink food-psychology papers behind a well-known retraction scandal and in Festinger's classic 1959 study. Every confirmed flag reran in a fresh, network-isolated sandbox, with no false positives among them. Alongside the deterministic flags, its reasoning layer surfaced 57 suspect claims out of 91 for human review, the overclaims and method problems that no arithmetic check would catch. We also ran it on our own published plasma-physics papers and were a bit humbled by what it found.