Policy-Adaptive Estimator Selection for Off-Policy Evaluation
This addresses the challenge of data-driven estimator selection for OPE, which is crucial for accurately evaluating counterfactual policies in offline settings, representing a novel but incremental contribution to the field.
The paper tackles the problem of selecting the most accurate estimator for off-policy evaluation (OPE) when ground-truth accuracy is unavailable, by developing a policy-adaptive method that uses subsampled logged data and pseudo policies, resulting in substantial improvements over non-adaptive heuristics in experiments on synthetic and real-world data.
Off-policy evaluation (OPE) aims to accurately evaluate the performance of counterfactual policies using only offline logged data. Although many estimators have been developed, there is no single estimator that dominates the others, because the estimators' accuracy can vary greatly depending on a given OPE task such as the evaluation policy, number of actions, and noise level. Thus, the data-driven estimator selection problem is becoming increasingly important and can have a significant impact on the accuracy of OPE. However, identifying the most accurate estimator using only the logged data is quite challenging because the ground-truth estimation accuracy of estimators is generally unavailable. This paper studies this challenging problem of estimator selection for OPE for the first time. In particular, we enable an estimator selection that is adaptive to a given OPE task, by appropriately subsampling available logged data and constructing pseudo policies useful for the underlying estimator selection task. Comprehensive experiments on both synthetic and real-world company data demonstrate that the proposed procedure substantially improves the estimator selection compared to a non-adaptive heuristic.