Doubly Robust Off-policy Value Evaluation for Reinforcement Learning
This addresses a critical step for applying RL in real-world problems by providing a more reliable method for policy evaluation, though it is an incremental extension of existing techniques.
The paper tackles off-policy value evaluation in reinforcement learning by extending the doubly robust estimator from bandits to sequential decision-making, achieving unbiasedness and lower variance than importance sampling methods, with demonstrations on benchmark problems and theoretical matching of lower bounds.
We study the problem of off-policy value evaluation in reinforcement learning (RL), where one aims to estimate the value of a new policy based on data collected by a different policy. This problem is often a critical step when applying RL in real-world problems. Despite its importance, existing general methods either have uncontrolled bias or suffer high variance. In this work, we extend the doubly robust estimator for bandits to sequential decision-making problems, which gets the best of both worlds: it is guaranteed to be unbiased and can have a much lower variance than the popular importance sampling estimators. We demonstrate the estimator's accuracy in several benchmark problems, and illustrate its use as a subroutine in safe policy improvement. We also provide theoretical results on the hardness of the problem, and show that our estimator can match the lower bound in certain scenarios.