Teaching LLMs to Recommend and Defer in Underrepresented Epilepsy Care
For frontline clinicians in resource-constrained settings, this work provides a method to adapt LLMs to local prescribing practice with a deferral mechanism, though the approach is evaluated on a specific domain and may be incremental.
The paper addresses the challenge of adapting LLMs for anti-seizure medication recommendation in Ugandan pediatric epilepsy care, where specialist expertise is scarce. The proposed MANANA framework improves top-3 prescription accuracy by 4-8 percentage points over baselines and enables selective prediction with 95% precision on the most confident half of cases.
Specialist epilepsy expertise is scarce in resource-constrained settings, making LLM-based decision support attractive for frontline clinicians managing longitudinal treatment. Such systems must adapt to local prescribing practice and know when to defer. We study this problem in Ugandan pediatric epilepsy care, predicting anti-seizure medication regimens from longitudinal unstructured clinic notes. Standard prompting achieves non-trivial agreement with physician prescriptions, but neurologist review shows that many errors reflect distribution-miscalibrated prescribing defaults rather than failures to parse the local record. We introduce MANANA, a non-parametric prompt-learning framework that learns local prescribing guidance from a small patient-level training set. MANANA converts observed prescription errors into auditable prompt memories, instantiated in single-agent and multi-agent variants, and improves over classical ML models, direct LLM prompting, and prompt-optimization baselines across two independently collected Ugandan cohorts. We further propose Bayesian prompt averaging, which converts the learned prompt trajectory into prescription likelihoods and an uncertainty-based deferral signal. On the independently collected held-out cohort, this improves visit-level top-3 prescription accuracy by 4-8 percentage points over prompt-optimization baselines and enables selective prediction: the system can auto-handle the most confident half of cases at 95% precision, or the most confident quarter at 99% precision, while deferring lower-confidence cases for specialist review.