confounded

The motivating question AI agents can now execute science at a level that was unthinkable two years ago. So the scarce resource has moved. It is no longer execution. It is asking a good question and exercising good judgment about what came back. That reframing is the whole project. If an agent will run any analysis you ask for and render it at publication quality, then the bottleneck — and the risk — sits entirely on the two ends: the question you posed, and your ability to judge the artifact it handed you. Confounded is a small, hostile test of whether anyone can actually do that second thing. --- What was built A figure factory. Real R, real DESeq2, real published data (the airway dataset — 4 donors × treated/untreated, paired). Every level starts from a named flaw, corrupts the pipeline in exactly that way, and lets it render something beautiful. Ground truth is free, because we planted it. The plot functions never learn whether the analysis behind them was sound — styling a figure by its truth would destroy the entire argument. A three-arm eval. Every figure is peer-reviewed by Opus, Sonnet and Haiku under three conditions. The picture is identical in all three; only what the reviewer knows about how it was made changes: - blind — the figure, methods caption stripped. What the pixels carry. - caption — the figure as a journal prints it. What peer review actually gets. - code — plus the prompt and the source that produced it. An app (confounded.vercel.app) https://github.com/sahuno/confounded that runs those arms on a human: judge the figure blind, commit an answer, then see the code, then watch your own verdict change — next to what the models said on the same figure.

Team

Placement

Participant