Generator-Mediated Bandits: Thompson Sampling for GenAI-Powered Adaptive Interventions
This work addresses the challenge of adapting bandit algorithms for GenAI-powered interventions, such as mobile health, by providing a novel method that leverages observed treatments for improved policy learning, representing a domain-specific advancement.
The paper tackles the problem of integrating generative AI models into personalized decision systems, where standard bandit methods fail to account for the split between actions and stochastic treatments, and introduces GAMBITTS, which explicitly models treatment and reward processes to accelerate learning, showing consistent outperformance over conventional algorithms in simulations.
Recent advances in generative artificial intelligence (GenAI) models have enabled the generation of personalized content that adapts to up-to-date user context. While personalized decision systems are often modeled using bandit formulations, the integration of GenAI introduces new structure into otherwise classical sequential learning problems. In GenAI-powered interventions, the agent selects a query, but the environment experiences a stochastic response drawn from the generative model. Standard bandit methods do not explicitly account for this structure, where actions influence rewards only through stochastic, observed treatments. We introduce generator-mediated bandit-Thompson sampling (GAMBITTS), a bandit approach designed for this action/treatment split, using mobile health interventions with large language model-generated text as a motivating case study. GAMBITTS explicitly models both the treatment and reward generation processes, using information in the delivered treatment to accelerate policy learning relative to standard methods. We establish regret bounds for GAMBITTS by decomposing sources of uncertainty in treatment and reward, identifying conditions where it achieves stronger guarantees than standard bandit approaches. In simulation studies, GAMBITTS consistently outperforms conventional algorithms by leveraging observed treatments to more accurately estimate expected rewards.