Shadi Shafighi

### What I built and investigated Regulatory-variant prediction models are commonly evaluated against massively parallel reporter assays, or MPRAs, as though the assay were direct ground truth. I built an estimand-aware benchmark audit that first asks a more fundamental question: do the model and experiment estimate the same biological quantity? I applied this framework to 3,976 psychiatric-risk regulatory variants studied in the developing human cortex. I compared three sequence-to-function models—AlphaGenome, Borzoi, and Enformer—with a genomic language model, Evo 2; a developing-cortex lentiMPRA; and a fine-mapped fetal-cortex eQTL atlas used as an external endogenous anchor. With Claude, I implemented custom Bayesian measurement-error and latent-variable models, coordinated model scoring, checked the analysis for reproducibility and overclaiming, and produced a calibrated per-variant concordance map. The latent agreement estimator was validated through simulation-based calibration. ### What I found The three activity-predicting models showed strong agreement with one another, with rank correlations of approximately 0.55–0.81, but essentially no agreement with the lentiMPRA, with correlation around −0.02. Accounting for the reporter’s reported measurement uncertainty did not restore agreement, indicating that this uncertainty alone is insufficient to explain the gap. Evo 2 did not join either group, showing that “sequence model” does not represent a single estimand. The three activity models also showed modest agreement with the fine-mapped eQTL effects, with correlations of approximately 0.24–0.28, while the reporter showed essentially none. Among the 25 variants with the strongest fine-mapping support, model and eQTL directions agreed for 20. ### Why it matters The conclusion is not that computational models beat experiments. It is that a precise experimental measurement may still be a misaligned benchmark when it targets a different biological quantity. Here, the lentiMPRA measures an isolated DNA element in an ectopically integrated construct, while the models predict endogenous functional signals from native-locus sequence. Their disagreement is therefore consistent with a consequential estimand mismatch, although native-locus editing is still required to distinguish this explanation from model error and shared bias. The result is not a model-versus-assay winner. It is a calibrated map showing where the measurements agree, where they diverge, and which native-locus experiments would be most informative next.

Team

Placement

Participant