Uncertainty-Aware Instance Reweighting for Off-Policy Learning
This work addresses a key bottleneck in off-policy learning for applications like recommender systems, offering an incremental improvement over existing methods.
The paper tackles the problem of high bias and variance in off-policy learning due to inaccurate logging policy estimates, proposing an Uncertainty-aware Inverse Propensity Score estimator (UIPS) that improves sample efficiency, as demonstrated on synthetic and real-world recommendation datasets.
Off-policy learning, referring to the procedure of policy optimization with access only to logged feedback data, has shown importance in various real-world applications, such as search engines, recommender systems, and etc. While the ground-truth logging policy, which generates the logged data, is usually unknown, previous work simply takes its estimated value in off-policy learning, ignoring both high bias and high variance resulted from such an estimator, especially on samples with small and inaccurately estimated logging probabilities. In this work, we explicitly model the uncertainty in the estimated logging policy and propose a Uncertainty-aware Inverse Propensity Score estimator (UIPS) for improved off-policy learning, with a theoretical convergence guarantee. Experiment results on synthetic and three real-world recommendation datasets demonstrate the advantageous sample efficiency of the proposed UIPS estimator against an extensive list of state-of-the-art baselines.