# GroundTruth

- **Event:** [Built with Claude: Life Sciences](https://cerebralvalley.ai/e/built-with-claude-life-sciences)
- **When:** Jul 7 at 12:00 PM – Jul 14 at 12:00 AM (EDT)
- **Where:** Online
- **Team:** [Victoria Wang](https://cerebralvalley.ai/u/VictoriaxWang)
- **GitHub:** https://github.com/victoriaxxwang/clinical-ai-eval-designer
- **Demo video:** https://youtu.be/znmxWjoSaGk
- **Gallery:** https://cerebralvalley.ai/e/built-with-claude-life-sciences/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/built-with-claude-life-sciences/hackathon/gallery/143

What I built. Clinical AI Eval Designer turns a clinical AI's intended use — the model, the patient population, the care setting, and the claim it makes — into a structured, citable validation specification: eight fields (study design, sensor/input validation, performance benchmarks, ground truth, sample size, subgroups, regulatory pathway, post-deployment monitoring), each with a recommendation, a cited rationale, a HIGH/MEDIUM/LOW confidence flag, and the expert sign-off it still needs.

It runs as one pipeline, three stages:
1) Retrieval — plain code, no model. It queries ClinicalTrials.gov, openFDA, and the literature before anything is written, so every citation is a re-resolvable identifier (PMID, NCT number, FDA K-number/product code) — verifiable by construction, not recalled from memory.
2) Synthesis — Claude writes the spec. Claude Fable 5 (Opus 4.8 fallback) maps the retrieved records into the eight fields under one hard rule: no invented numbers — cite it, or flag it for the study team to set.
3) Review — a three-persona panel. An optional pass has three Claude-played reviewers (regulator, biostatistician, clinical scientist) critique the spec under the same cite-or-flag discipline.

What I found. Trustworthy output takes engineered discipline at two boundaries, not one. First, retrieval has to be deterministic. The moment the model fetches its own evidence, the same input returns slightly different sources run to run — and those differences cascade into everything the spec cites; my ablation runs showed recall failures degrade the whole downstream result. So retrieval is plain code with re-resolvable identifiers, reproducible by construction. Second, deterministic retrieval still isn't safe on its own: hand an unconstrained model even perfect evidence and it produces confident, fabricated benchmarks. So synthesis runs under a constraint layer — cite-or-flag, confidence tiers, and naming-and-rejecting off-target records instead of citing them to look thorough. Guard only one boundary and the spec looks authoritative while being subtly false. Guard both, and weeks of expert literature work compress into a grounded starting point in minutes — while staying honest about what still needs a human.

Why it matters. A clinical AI can't reach the patients it's meant to help until it clears the FDA — a slow process with no single playbook that fits every model, where the right validation evidence changes by indication, population, and claim. A model can perform well in the lab and still stall on the way to the clinic, because assembling that evidence is slow, manual, and specific to each case. This tool turns weeks of that groundwork into a rigorous, grounded starting point — and an expert-review conversation — with every claim tied to a source the team can re-check. The principle it demonstrates is the one the demo closes on: verify, don't blindly trust — honesty enforced in how the model is allowed to reason, not a surface guardrail: a "not medical advice" disclaimer or warning banner bolted onto the interface while the model underneath could still fabricate.

How I validated it. I built a hand-verified answer key for 10 diverse clinical AI cases (devices, drugs, biologics; regulatory-approved and regulatory-null) and scored the live pipeline on precision and recall — never raw counts. To make retrieval trustworthy I ran an ablation study — varying one component at a time across all 10 cases to confirm each earned its place and to lock the best default — including a disease-aware search that recognizes the condition even when the input phrases it indirectly, and a bridge from that condition to the matching FDA product codes and cleared devices. Every one of the ~470 scored identifiers re-resolves against its live registry today, and across the slate the tool cites real cleared devices where they exist and names-and-rejects off-target records everywhere else — for zero fabricated predicates. (Full method: eval_results/ablation_findings.md.)

---

Markdown version of https://cerebralvalley.ai/e/built-with-claude-life-sciences/hackathon/gallery/143. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt