PhysTax

PhysTax: an AI scientist that sees only through SAM 3.1, seals its predictions before reality answers, and is graded by what actually happens. It measures SAM 3.1 as a instrument and holds Muse Spark to sealed predictions across simulated physics, real video, and an open problem from Meta's own research. Problem: AI-scientist benchmarks hand agents clean numbers, but real science, robotics and world-model evaluation measure through vision models. Nobody knows what those perception errors cost scientific conclusions, or whether the agent's reasoning is the real bottleneck. Method: Muse Spark experiments in universes with hidden, non-textbook force laws (orbit a = -k r^-alpha, projectile drag |v|^beta, nonlinear springs). It sees experiments only through SAM 3.1 tracking, fits laws in a sandbox, then seals a prediction (law, constants, 80% intervals) as a SHA-256 hash in a hash-chained ledger before the test exists. A deterministic grader scores it; no LLM judges. Since the truth is known, every error splits exactly into measurement (camera) + design (experiments) + reasoning (law chosen). Results (reproducible from cached runs): 1. The camera is not the bottleneck, reasoning is: over 18 Muse Spark episodes, through SAM the error is 0.009 measurement, 0.000 design, 0.226 reasoning (camera about 3%). 2. The agent's sealed law carries real physics (CMU Predictive Credit protocol): a fresh forecaster's error falls from 49.8% to 19.0% with it (true law: 3.6%); credit +20.4 pts, 95% CI 3.3-28.4. 3. SAM 3.1 as an instrument (432 tracks): 0.7-2.1 px interior error vs 85 px classical; 5-8x worse at borders; names found 4/12 times, taps/boxes always; box confidence predicts error (Spearman -0.61, 10,685 frames); correlated errors shrink textbook 80% interval coverage to 17-25% (our correction: 46-58%). 4. Open problem from Meta's research: Arslan & Muse Spark's ellipsoid-fitting theorem leaves the edge n/d^2 -> 1/4 open. We measured it by semidefinite programming and sealed four predictions before computing d = 30, 36, 44: Muse Spark's hit the 95% interval 3/3; our own conjecture d(d+1)/4 was falsified 3/3. Finite-size evidence, not a proof. 5. Real footage: on 7 IRIS benchmark pendulum clips (ECCV 2026), the exponential damping its ground truth assumes is rejected 7/7. A live webcam mode lets anyone tap an object; the scientist names the motion and seals the second half at a split fixed in advance. Engineering: hash-chained ledgers, cached replayable SAM calls, budget guards, 40 CI tests, public results site, documented limitations, under $10 API spend. Frontier: a first measured error budget for a segmentation foundation model used as a scientific instrument, with AI scientists held to falsifiable, pre-registered predictions across physics, real video and open mathematics. Next: verifiable-reward RL environments for scientist agents, error budgets for SAM-graded world-model benchmarks, open-weight Glimmer scientists, and real lab instruments.