Doubly-Robust Off-Policy Evaluation with Estimated Logging Policy
This work addresses a key challenge in reinforcement learning and contextual bandits for researchers and practitioners, offering an incremental improvement in estimator efficiency.
The authors tackled the problem of off-policy evaluation when both the logging policy and value function are unknown by introducing a novel doubly-robust estimator called DRUnknown, which achieves the smallest asymptotic variance under correct model specifications and reaches the semiparametric lower bound when both models are correct.
We introduce a novel doubly-robust (DR) off-policy evaluation (OPE) estimator for Markov decision processes, DRUnknown, designed for situations where both the logging policy and the value function are unknown. The proposed estimator initially estimates the logging policy and then estimates the value function model by minimizing the asymptotic variance of the estimator while considering the estimating effect of the logging policy. When the logging policy model is correctly specified, DRUnknown achieves the smallest asymptotic variance within the class containing existing OPE estimators. When the value function model is also correctly specified, DRUnknown is optimal as its asymptotic variance reaches the semiparametric lower bound. We present experimental results conducted in contextual bandits and reinforcement learning to compare the performance of DRUnknown with that of existing methods.