Vitalii Babenko

ClinVal Validator is built for one named user: the clinical-AI researcher who has trained a medical-imaging model and needs to know whether the reported AUC is real or an artifact of how the data was split. It ingests a model's predictions.csv (predictions plus patient/scanner/site metadata) and runs three deterministic checks: patient-level leakage (do patient IDs cross the train/test boundary?), a split-quality audit (re-score the same predictions on a patient-disjoint subset), and per-subgroup degradation (AUC by scanner and site). It then hands the resulting JSON to Claude, which writes either a GREEN/YELLOW/RED risk report or a journal-reviewer critique with a corrected, leakage-safe Methods paragraph the author can paste into their manuscript. One architectural rule is enforced end to end: Python computes every number; Claude only explains it. The validators run fully offline; the reporter is the only code that touches the network, and it may cite only figures already present in the JSON. It never computes a metric. What I found: on the real RSNA Screening Mammography cohort, a naive row-level random split — the default most people reach for — leaves 7,511 of 7,524 test patients (99.8%) also present in training, because each patient contributes four to five images. Nothing is wrong with RSNA. The mistake is splitting by image instead of by patient, and it is completely invisible unless someone counts. Why it matters: a leaked split doesn't merely inflate a number — it makes the number unanswerable, because it no longer measures what a clinical reader assumes it measures. The detectors are verified against six ground-truth-labeled synthetic scenarios, so the tool's own claims are checked, not asserted. MIT, no PHI, synthetic/public-schema data only.

Team

Placement

Participant