Bootstrapping Fitted Q-Evaluation for Off-Policy Inference
This work provides a theoretically sound and computationally efficient method for assessing the quality and uncertainty of batch reinforcement learning policies, which is crucial for practitioners deploying RL systems.
This paper proposes a bootstrapping Fitted Q-Evaluation (FQE) method to infer the distribution of policy evaluation error in off-policy evaluation. The method is shown to be asymptotically efficient and distributionally consistent for off-policy statistical inference, with a subsampling procedure improving runtime by an order of magnitude.
Bootstrapping provides a flexible and effective approach for assessing the quality of batch reinforcement learning, yet its theoretical property is less understood. In this paper, we study the use of bootstrapping in off-policy evaluation (OPE), and in particular, we focus on the fitted Q-evaluation (FQE) that is known to be minimax-optimal in the tabular and linear-model cases. We propose a bootstrapping FQE method for inferring the distribution of the policy evaluation error and show that this method is asymptotically efficient and distributionally consistent for off-policy statistical inference. To overcome the computation limit of bootstrapping, we further adapt a subsampling procedure that improves the runtime by an order of magnitude. We numerically evaluate the bootrapping method in classical RL environments for confidence interval estimation, estimating the variance of off-policy evaluator, and estimating the correlation between multiple off-policy evaluators.