Data Alchemist

Data Alchemist is a flight recorder for AI data analysts. Ask an AI agent "what drives our sales?" and it will run dozens of statistical tests, then show you the two or three that look impressive. You never see the rest. At the usual 5% threshold, 14 hidden tests mean a 51% chance of at least one false finding; 45 mean 90%. We measured it on Muse Spark. On 5 synthetic datasets with no real effect, it made 56 claims: 39 correct denials, and 17 asserting a link that does not exist. Nothing in its answer says which. Data Alchemist sits between the agent and your data: - Every test is recorded. The agent can only get a p-value through our API, so we always know how many tests were run. Unregistered or altered extractions are refused. - The report is written by the engine, never by the agent: tests run, what looked significant, what was confirmed, and every rejected claim with the reason. - Promising findings are rechecked once on data the agent never saw. The number of hypotheses to confirm is fixed before that second half is opened, and it cannot be opened twice. - Multiple-testing corrections (Bonferroni, Holm, BH, BY, e-BH) are built in, with their assumptions stated. On the same noise, the engine kept none of the 17 false claims. That is the easy direction, so we are also measuring the hard one: whether real effects survive, on planted relationships with confounders. We pre-registered our protocol and fixed the decision rule in advance: we say "p-hacking" only if registry p-values are uniform while the reported ones pile up below 5%. Otherwise we say over-claiming. We would rather publish an unflattering result than an unsupported headline. Muse Spark is the first model we audit, through an adapter; the engine itself is model-agnostic. The code is open source, with automated tests on the core.