Balanced off-policy evaluation in general action spaces
This work addresses a specific technical bottleneck in offline policy evaluation for contextual bandits, offering an incremental improvement over existing weighting-based approaches.
The paper tackles the problem of imbalance in importance sampling weights for off-policy evaluation in contextual bandits, presenting Balanced Off-Policy Evaluation (B-OPE) as a method that reduces this imbalance by framing it as a binary classification problem, with experimental results showing improvements in both discrete and continuous action spaces.
Estimation of importance sampling weights for off-policy evaluation of contextual bandits often results in imbalance - a mismatch between the desired and the actual distribution of state-action pairs after weighting. In this work we present balanced off-policy evaluation (B-OPE), a generic method for estimating weights which minimize this imbalance. Estimation of these weights reduces to a binary classification problem regardless of action type. We show that minimizing the risk of the classifier implies minimization of imbalance to the desired counterfactual distribution of state-action pairs. The classifier loss is tied to the error of the off-policy estimate, allowing for easy tuning of hyperparameters. We provide experimental evidence that B-OPE improves weighting-based approaches for offline policy evaluation in both discrete and continuous action spaces.