High-confidence error estimates for learned value functions
This addresses a largely open problem for reinforcement learning researchers needing reliable accuracy assessments of policy evaluation algorithms, though it is incremental as it builds on existing methods and highlights remaining challenges.
The paper tackles the problem of obtaining high-confidence error estimates for learned value functions in reinforcement learning, particularly for large or continuous state-spaces where true values are infeasible to compute, and provides a bound and offline sampling algorithm with experimental investigation into sample requirements.
Estimating the value function for a fixed policy is a fundamental problem in reinforcement learning. Policy evaluation algorithms---to estimate value functions---continue to be developed, to improve convergence rates, improve stability and handle variability, particularly for off-policy learning. To understand the properties of these algorithms, the experimenter needs high-confidence estimates of the accuracy of the learned value functions. For environments with small, finite state-spaces, like chains, the true value function can be easily computed, to compute accuracy. For large, or continuous state-spaces, however, this is no longer feasible. In this paper, we address the largely open problem of how to obtain these high-confidence estimates, for general state-spaces. We provide a high-confidence bound on an empirical estimate of the value error to the true value error. We use this bound to design an offline sampling algorithm, which stores the required quantities to repeatedly compute value error estimates for any learned value function. We provide experiments investigating the number of samples required by this offline algorithm in simple benchmark reinforcement learning domains, and highlight that there are still many open questions to be solved for this important problem.