Unbiased Estimation of the Value of an Optimized Policy
This addresses the need for reliable policy evaluation in high-risk scenarios like A/B testing, offering a method to reduce bias in estimates, though it is incremental as it builds on existing bagging and estimation techniques.
The paper tackles the problem of obtaining unbiased estimates for the value of an optimized policy learned from A/B test data, where direct use of samples leads to bias. It presents a bagging-based procedure that provides unbiased estimators, such as inverse-propensity-weighting and doubly-robust estimators, and empirically demonstrates that positive optimized policies can be found even with negative average treatment effects.
Randomized trials, also known as A/B tests, are used to select between two policies: a control and a treatment. Given a corresponding set of features, we can ideally learn an optimized policy P that maps the A/B test data features to action space and optimizes reward. However, although A/B testing provides an unbiased estimator for the value of deploying B (i.e., switching from policy A to B), direct application of those samples to learn the the optimized policy P generally does not provide an unbiased estimator of the value of P as the samples were observed when constructing P. In situations where the cost and risks associated of deploying a policy are high, such an unbiased estimator is highly desirable. We present a procedure for learning optimized policies and getting unbiased estimates for the value of deploying them. We wrap any policy learning procedure with a bagging process and obtain out-of-bag policy inclusion decisions for each sample. We then prove that inverse-propensity-weighting effect estimator is unbiased when applied to the optimized subset. Likewise, we apply the same idea to obtain out-of-bag unbiased per-sample value estimate of the measurement that is independent of the randomized treatment, and use these estimates to build an unbiased doubly-robust effect estimator. Lastly, we empirically shown that even when the average treatment effect is negative we can find a positive optimized policy.