Distributionally Robust Policy Evaluation under General Covariate Shift in Contextual Bandits
This addresses reliability issues in offline policy evaluation for contextual bandits, which is important for applications like recommendation systems and healthcare, though it appears incremental as it builds on existing evaluation frameworks.
The paper tackles the problem of offline policy evaluation in contextual bandits under general covariate shifts, where discrepancies exist between logging and target data distributions. The result is a distributionally robust approach that significantly outperforms baseline methods in 90% of policy shift-only cases and 72% of general covariate shift scenarios.
We introduce a distributionally robust approach that enhances the reliability of offline policy evaluation in contextual bandits under general covariate shifts. Our method aims to deliver robust policy evaluation results in the presence of discrepancies in both context and policy distribution between logging and target data. Central to our methodology is the application of robust regression, a distributionally robust technique tailored here to improve the estimation of conditional reward distribution from logging data. Utilizing the reward model obtained from robust regression, we develop a comprehensive suite of policy value estimators, by integrating our reward model into established evaluation frameworks, namely direct methods and doubly robust methods. Through theoretical analysis, we further establish that the proposed policy value estimators offer a finite sample upper bound for the bias, providing a clear advantage over traditional methods, especially when the shift is large. Finally, we designed an extensive range of policy evaluation scenarios, covering diverse magnitudes of shifts and a spectrum of logging and target policies. Our empirical results indicate that our approach significantly outperforms baseline methods, most notably in 90% of the cases under the policy shift-only settings and 72% of the scenarios under the general covariate shift settings.