CiteEval
Built with Claude: Life Sciencesciteval is a citation referee for the life-sciences literature. Given one scientific claim and the paper cited to support it, it decides whether the paper actually supports the claim — across three axes, each verdict carrying a confidence tier and a provenance label (metadata-grounded vs. model-inferred): (1) retraction, checked against a ~29k-record Retraction Watch index cross-referenced with NCBI; (2) species/population scope, where MeSH organism/age tags establish the paper's real scope and Claude Opus 4.8 judges whether the claim over-generalizes; and (3) a six-level substantive-support grade from an adversarial supporter→refuter→critic triad on Opus 4.8. The guiding principle is grounded-and-measured, not model-checks-model: objective facts are decided by external structured records that override the model, and the subjective grade is measured against an independent reference, calibrated, and stress-tested on a held-out set it was never tuned on. What we found: retraction detection is perfect in-domain and out-of-sample (P/R 1.00); the graded axis reaches 0.90 quadratic-weighted kappa, is within one level on every case, carries zero confident-but-wrong grades, and recalibrates to an honest 0.08 ECE; scope holds at 0.909 in-domain / 0.71 held-out. Along the way the tool overruled its own answer key (flagging retractions our corpus had mislabeled), and a held-out test caught us propping a scope metric — which we fixed honestly rather than hide. Why it matters: a 2025 review puts the quotation-error rate in the medical literature at 16.9%, retracted papers keep being cited as live, and AI drafting is spreading misattribution. citeval answers the specific, checkable question — does THIS paper support THIS claim? — and refuses to answer with a number it can't defend. It ships with a live web UI (five-stage pipeline stepper, adversarial-debate replay, live cost/ETA, parallel multi-paper dashboard), a FastAPI backend, and offline proof pages — every headline number reproducible offline in seconds via make eval.
Team
Placement
Participant
