# CiteEval

- **Event:** [Built with Claude: Life Sciences](https://cerebralvalley.ai/e/built-with-claude-life-sciences)
- **When:** Jul 7 at 12:00 PM – Jul 14 at 12:00 AM (EDT)
- **Where:** Online
- **Team:** [Pranav Lakhwara](https://cerebralvalley.ai/u/Mythical_Hacker)
- **GitHub:** https://github.com/mythicalhacker/citeval
- **Demo video:** https://youtu.be/5Vk0iH2gO6Y
- **Gallery:** https://cerebralvalley.ai/e/built-with-claude-life-sciences/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/built-with-claude-life-sciences/hackathon/gallery/253

citeval is a citation referee for the life-sciences literature. Given one scientific claim and the paper cited to support it, it decides whether the paper actually supports the claim — across three axes, each verdict carrying a confidence tier and a provenance label (metadata-grounded vs. model-inferred): (1) retraction, checked against a ~29k-record Retraction Watch index cross-referenced with NCBI; (2) species/population scope, where MeSH organism/age tags establish the paper's real scope and Claude Opus 4.8 judges whether the claim over-generalizes; and (3) a six-level substantive-support grade from an adversarial supporter→refuter→critic triad on Opus 4.8.

The guiding principle is grounded-and-measured, not model-checks-model: objective facts are decided by external structured records that override the model, and the subjective grade is measured against an independent reference, calibrated, and stress-tested on a held-out set it was never tuned on.

What we found: retraction detection is perfect in-domain and out-of-sample (P/R 1.00); the graded axis reaches 0.90 quadratic-weighted kappa, is within one level on every case, carries zero confident-but-wrong grades, and recalibrates to an honest 0.08 ECE; scope holds at 0.909 in-domain / 0.71 held-out. Along the way the tool overruled its own answer key (flagging retractions our corpus had mislabeled), and a held-out test caught us propping a scope metric — which we fixed honestly rather than hide.

Why it matters: a 2025 review puts the quotation-error rate in the medical literature at 16.9%, retracted papers keep being cited as live, and AI drafting is spreading misattribution. citeval answers the specific, checkable question — does THIS paper support THIS claim? — and refuses to answer with a number it can't defend. It ships with a live web UI (five-stage pipeline stepper, adversarial-debate replay, live cost/ETA, parallel multi-paper dashboard), a FastAPI backend, and offline proof pages — every headline number reproducible offline in seconds via make eval.

---

Markdown version of https://cerebralvalley.ai/e/built-with-claude-life-sciences/hackathon/gallery/253. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt