Uncertainty Sets for Distributionally Robust Bandits Using Structural Equation Models
This work addresses the issue of conservatism in distributionally robust bandits for researchers and practitioners in reinforcement learning, representing an incremental improvement over traditional methods.
The authors tackled the problem of overly conservative evaluations and policies in distributionally robust bandits by proposing a practical algorithm that tailors uncertainty sets using structural equation models, achieving more accurate evaluations and lower-variance policies, especially for large shifts.
Distributionally robust evaluation estimates the worst-case expected return over an uncertainty set of possible covariate and reward distributions, and distributionally robust learning finds a policy that maximizes that worst-case return across that uncertainty set. Unfortunately, current methods for distributionally robust evaluation and learning create overly conservative evaluations and policies. In this work, we propose a practical bandit evaluation and learning algorithm that tailors the uncertainty set to specific problems using mathematical programs constrained by structural equation models. Further, we show how conditional independence testing can be used to detect shifted variables for modeling. We find that the structural equation model (SEM) approach gives more accurate evaluations and learns lower-variance policies than traditional approaches, particularly for large shifts. Further, the SEM approach learns an optimal policy, assuming the model is sufficiently well-specified.