TrustCheck
Built with Claude: Life SciencesProblem. A biologist generates new data and wants to build on the published literature, increasingly with an AI scientist running the analysis. That raises two questions: are the findings they build on reproducible in their data, and is the AI scientist reliable? Solution. The biologist runs TrustCheck: they name a finding, point it at their cohorts, and it computes, live on the raw data, whether it holds. The verdict is that number, not the AI scientist's judgment, and the AI scientist works blind, never told which finding is supposed to reproduce, so it cannot just confirm what is expected. When a finding fails, two follow-up checks, power and method, explain why. Example. We tested two discoveries from our 2023 Nature Medicine paper on single-cell data from immunotherapy-treated liver (HCC) tumors, against two independent cohorts. One reproduced in all three. The other, testable in only one of the two, reproduced in HCC but not in melanoma, where 100% power and 0 of 20 analyses rule out both a small cohort and a bad pipeline. TrustCheck does not call the finding false: an effect as large as the original would have shown up here, but a smaller one cannot be excluded.
