Confident Off-Policy Evaluation and Selection through Self-Normalized Importance Weighting
This addresses the need for reliable policy evaluation in bandit settings, offering an incremental improvement over existing methods for practitioners in reinforcement learning.
The paper tackles the problem of robust off-policy selection in contextual bandits by proposing a method to compute a lower bound on policy value with desired coverage, using Self-normalized Importance Weighting. The method outperforms competitors in synthetic and real datasets, providing tighter confidence intervals and better policy choices.
We consider off-policy evaluation in the contextual bandit setting for the purpose of obtaining a robust off-policy selection strategy, where the selection strategy is evaluated based on the value of the chosen policy in a set of proposal (target) policies. We propose a new method to compute a lower bound on the value of an arbitrary target policy given some logged data in contextual bandits for a desired coverage. The lower bound is built around the so-called Self-normalized Importance Weighting (SN) estimator. It combines the use of a semi-empirical Efron-Stein tail inequality to control the concentration and a new multiplicative (rather than additive) control of the bias. The new approach is evaluated on a number of synthetic and real datasets and is found to be superior to its main competitors, both in terms of tightness of the confidence intervals and the quality of the policies chosen.