On the Reuse Bias in Off-Policy Reinforcement Learning
This addresses a specific instability problem in off-policy RL for researchers and practitioners, offering an incremental improvement over existing variance-focused methods.
The paper tackles instability in off-policy reinforcement learning by identifying a new 'Reuse Bias' in importance sampling, where reusing the replay buffer for evaluation and optimization causes overestimation, and proposes a Bias-Regularized Importance Sampling (BIRIS) framework that significantly improves sample efficiency on MuJoCo continuous control tasks.
Importance sampling (IS) is a popular technique in off-policy evaluation, which re-weights the return of trajectories in the replay buffer to boost sample efficiency. However, training with IS can be unstable and previous attempts to address this issue mainly focus on analyzing the variance of IS. In this paper, we reveal that the instability is also related to a new notion of Reuse Bias of IS -- the bias in off-policy evaluation caused by the reuse of the replay buffer for evaluation and optimization. We theoretically show that the off-policy evaluation and optimization of the current policy with the data from the replay buffer result in an overestimation of the objective, which may cause an erroneous gradient update and degenerate the performance. We further provide a high-probability upper bound of the Reuse Bias, and show that controlling one term of the upper bound can control the Reuse Bias by introducing the concept of stability for off-policy algorithms. Based on these analyses, we finally present a novel Bias-Regularized Importance Sampling (BIRIS) framework along with practical algorithms, which can alleviate the negative impact of the Reuse Bias. Experimental results show that our BIRIS-based methods can significantly improve the sample efficiency on a series of continuous control tasks in MuJoCo.