Context in Public Health for Underserved Communities: A Bayesian Approach to Online Restless Bandits
This addresses resource allocation in public health for underserved communities, but it is incremental as it builds on existing restless bandit and Bayesian methods.
The paper tackles the problem of allocating limited interventions in public health programs for underserved communities by modeling it as a restless multi-armed bandit with unknown dynamics, and presents BCoR, a Bayesian online reinforcement learning approach that achieves substantially higher finite-sample performance in experiments, including with real-world maternal mHealth data from India.
Public health programs often provide interventions to encourage program adherence, and effectively allocating interventions is vital for producing the greatest overall health outcomes, especially in underserved communities where resources are limited. Such resource allocation problems are often modeled as restless multi-armed bandits (RMABs) with unknown underlying transition dynamics, hence requiring online reinforcement learning (RL). We present Bayesian Learning for Contextual RMABs (BCoR), an online RL approach for RMABs that novelly combines techniques in Bayesian modeling with Thompson sampling to flexibly model the complex RMAB settings present in public health program adherence problems, namely context and non-stationarity. BCoR's key strength is the ability to leverage shared information within and between arms to learn the unknown RMAB transition dynamics quickly in intervention-scarce settings with relatively short time horizons, which is common in public health applications. Empirically, BCoR achieves substantially higher finite-sample performance over a range of experimental settings, including a setting using real-world adherence data that was developed in collaboration with ARMMAN, an NGO in India which runs a large-scale maternal mHealth program, showcasing BCoR practical utility and potential for real-world deployment.