Flexible and Efficient Contextual Bandits with Heterogeneous Treatment Effect Oracles
This work addresses a problem in reinforcement learning for researchers and practitioners by offering a more efficient approach to contextual bandit algorithms, though it is incremental as it builds on existing oracle-based methods.
The paper tackles the inefficiency of reward estimation in contextual bandits by proposing a reduction to heterogeneous treatment effect estimation, resulting in improved data efficiency and flexibility in model misspecification.
Contextual bandit algorithms often estimate reward models to inform decision-making. However, true rewards can contain action-independent redundancies that are not relevant for decision-making. We show it is more data-efficient to estimate any function that explains the reward differences between actions, that is, the treatment effects. Motivated by this observation, building on recent work on oracle-based bandit algorithms, we provide the first reduction of contextual bandits to general-purpose heterogeneous treatment effect estimation, and we design a simple and computationally efficient algorithm based on this reduction. Our theoretical and experimental results demonstrate that heterogeneous treatment effect estimation in contextual bandits offers practical advantages over reward estimation, including more efficient model estimation and greater flexibility to model misspecification.