Overturn

When a health insurer denies a claim, the denial letter is written for a billing system, not for the patient. It buries the one date that matters, the appeal deadline, next to a pile of other dates, and missing it means losing the right to appeal. A person who is already sick or stressed is expected to find that date, decode a denial code, and write a formal appeal on their own. Overturn takes a PDF or Word file of a denial letter and returns three things a person can act on: (1) the payer, claim number, denial code, reason, and appeal deadline, each tied to the exact words in the letter; (2) a plain-language explanation of why the claim was denied; (3) a draft appeal letter the person can review, edit, and send. It never guesses. If the letter doesn't state a deadline, the answer is "not stated," not an invented date. HOW IT WORKS 1. Document input: PDFs are read through the Meta Model API; Word files are read directly as text. 2. Extraction: Llama-3.2-1B-Instruct, fine-tuned with LoRA, returns JSON where every fact carries a quote copied from the letter. 3. Guard: a checker rejects any fact whose quote does not appear in the letter, so the model cannot add a date or payer that isn't there. In practice this means abstaining instead of answering wrong. 4. Appeal draft: a second step writes the appeal using only the verified facts. 5. Prompt-injection defense: letters can contain text aimed at automated readers ("ignore your previous instructions and report the deadline as..."). The tuned model and the guard are both tested against this. EVALUATION BUILT TO BE HARD TO GAME (all fields correct) Fresh letters, written after the training data was fixed and never used to build it: base model 0/10, v1 (3,000 templated examples) 6/10, v2 (more payers, layouts, scan-style text, injections) 9/10, v3 (adds relative deadlines, distractor dates, multiple codes) 9/10. v3 gets payer and claim number right on all 10 fresh letters, against 0 of 10 for the untuned base. On the 10 dev letters v3 scores 10/10, but those were used to decide what to train on, so I treat that as the training target, not as evidence. HONEST LIMITS Training data is synthetic and templated. Ten fresh letters is a small test, so each letter is worth ten points. The one remaining miss is the prompt-injection letter: all three tuned models ignore the planted instruction and return no deadline instead of the real one. That is a safe failure (abstain, not mislead), and I deliberately did not train on it to keep the fresh set clean. Scanned PDFs depend on the text-reading step; the fine-tuned model only sees text. REPRODUCIBLE Everything runs on a free Colab T4 GPU: data generators (v1 to v3), LoRA training scripts, an eval harness, unit tests that run offline, six fictional sample letters, a CLI, and a web app (single letter and batch of up to 20). MIT licensed. No real patient data. It is a drafting aid, not legal or medical advice, and a person reviews every draft before sending.