Provenance Gate (solo team)

I investigated a failure mode of agentic science: as an agent forks approaches, revises an upstream step, and re-runs, a final result can quietly come to rest on stale or version-conflicting inputs. The conflict lives in the provenance, not the prose — so reading the output, a human or an LLM reviewer may have nothing to go on. To study it in Claude Science I built two instruments: a deterministic, read-only provenance gate that computes two structural verdicts per computation (a stale input; a lineage that reconverges on two live versions of one artifact), and a first-prototype harness that runs a scenario unattended and captures each rollout for grading. The finding: across 24 unattended rollouts of one PBMC single-cell scenario, 12 shipped a version-mixed figure and 6 shipped it with no warning anywhere in the conversation. Installing the gate as a skill wasn't enough for an agent to use it on its own; one line of standing policy was, and the agent then caught the conflict and stopped to ask. Yet handed the gate's flag directly, an agent still rationalized a real conflict as "benign" — which is exactly why the check must be deterministic: the verdict is read from the lineage, not the model's judgment. Why it matters: fast agentic science is only trustworthy if results trace to consistent inputs — a structural fact a deterministic check can hold even when the prose can't, and a natural candidate for a verifiable training signal, not just a review. Design rationale, limitations, and full rollout results are in the repo.

Team

Placement

Participant