BALL-RL

I built BALL-RL: Bayesian Anchored Latent Learning for Reinforcement Learning, a framework for learning personalized treatment policies when the clinically meaningful patient state is measured only intermittently. Our application focuses on repetitive transcranial magnetic stimulation (rTMS), in which clinicians repeatedly select from protocol, intensity, and laterality options, but validated symptom questionnaires are collected only a few times during a treatment course. BALL-RL uses these sparse questionnaires as clinical anchors for a Bayesian teacher-student decoder that reconstructs a continuous, forward-only severity trajectory from dense clinical records. That trajectory provides both the patient state and the session-level reward needed to train a conservative offline RL policy. I evaluated the complete system in a digital twin with known patient states, optimal actions, and counterfactual outcomes. BALL-RL recovered latent severity accurately and enabled the learned policy to close 97.2% of the performance gap between a naive sparse-questionnaire learner and a finite-sample oracle trained with the true latent reward. The benefit remained when BALL-RL was restricted to the same number of training transitions as the sparse comparator, demonstrating that the improvement came from the reward's denser, less biased temporal decomposition rather than simply from having more observations. I then applied the framework to 1,758 observational rTMS courses from 1,508 patients across 15 facilities, using patient-grouped, facility-held-out evaluation. Both fitted-Q evaluation and instrumented off-policy evaluation favored the learned recommendations over observed clinical decisions. AT-OPE estimated 0.350 latent improvement units per course left unrealized when care departed from the learned policy, equivalent to approximately 51.5% of the cohort’s average decoded course improvement. Finally, I built a clinician-facing synthetic-data demonstration with Claude integrated as an interactive chatbot for interpreting the RL model’s decisions. The interface presents the decoded patient trajectory, uncertainty, recommended protocol, supported alternatives, safety status, and SHAP-based drivers. Clinicians can ask Claude why one protocol was favored, why another was not selected, which patient-state features influenced the decision, or how uncertainty affected the result. Claude answers using a tightly constrained payload of model outputs, translating Q-values, feature attributions, and latent-state estimates into understandable clinical language. It does not select, modify, or rerank treatments: the verified RL policy makes the decision, while Claude helps the clinician understand and interrogate its reasoning. FHIR and CDS Hooks interfaces additionally demonstrate how this explainable decision-support workflow could integrate with an EHR. This matters because sparse outcomes are the rule rather than the exception in health care. Most clinical AI predicts a single outcome from a snapshot, but treatment is sequential: clinicians act, observe an incomplete response, and adapt. Without a reliable session-level state, conventional RL cannot determine which action produced improvement or learn when a treatment should be continued, intensified, switched, or stopped. BALL-RL addresses that missing layer by converting occasional validated measurements and routinely collected clinical records into a dense, uncertainty-aware state while preserving the clinical meaning supplied by the questionnaires. The result is a practical bridge between real-world clinical data and sequential decision optimization. It allows health systems to learn from treatment courses that have already occurred, without assigning exploratory treatments to patients, and produces recommendations that can be inspected, explained, support-constrained, and withheld when safety concerns arise. In rTMS, this creates a path to adapt protocol choices throughout a course rather than applying a largely fixed strategy and waiting weeks to evaluate the results. More broadly, the same problem arises wherever meaningful outcomes are delayed or intermittent, including medication titration, rehabilitation, chronic disease management, intensive care, and behavioral health. BALL-RL provides a general framework for turning sparse endpoints into actionable learning signals, enabling personalized treatment sequences in settings where existing measurement practices previously made them inaccessible.