Optimal Baseline Corrections for Off-Policy Contextual Bandits
This work addresses variance reduction in off-policy contextual bandits, which is crucial for applications like recommender systems, but it is incremental as it builds on existing control variate methods.
The paper tackles the high variance problem in off-policy learning for recommender systems by unifying control variate methods into a single framework, deriving an optimal unbiased estimator that significantly improves performance and reduces data requirements.
The off-policy learning paradigm allows for recommender systems and general ranking applications to be framed as decision-making problems, where we aim to learn decision policies that optimize an unbiased offline estimate of an online reward metric. With unbiasedness comes potentially high variance, and prevalent methods exist to reduce estimation variance. These methods typically make use of control variates, either additive (i.e., baseline corrections or doubly robust methods) or multiplicative (i.e., self-normalisation). Our work unifies these approaches by proposing a single framework built on their equivalence in learning scenarios. The foundation of our framework is the derivation of an equivalent baseline correction for all of the existing control variates. Consequently, our framework enables us to characterize the variance-optimal unbiased estimator and provide a closed-form solution for it. This optimal estimator brings significantly improved performance in both evaluation and learning, and minimizes data requirements. Empirical observations corroborate our theoretical findings.