Variance Reduction Based Experience Replay for Policy Optimization
This addresses the challenge of sample efficiency in reinforcement learning for complex stochastic systems, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackled the problem of inefficient experience replay in reinforcement learning by proposing Variance Reduction Experience Replay (VRER), which selectively reuses informative samples to reduce variance in policy gradient estimation, resulting in accelerated policy learning and improved performance over state-of-the-art algorithms.
Effective reinforcement learning (RL) for complex stochastic systems requires leveraging historical data collected in previous iterations to accelerate policy optimization. Classical experience replay treats all past observations uniformly and fails to account for their varying contributions to learning. To overcome this limitation, we propose Variance Reduction Experience Replay (VRER), a principled framework that selectively reuses informative samples to reduce variance in policy gradient estimation. VRER is algorithm-agnostic and integrates seamlessly with existing policy optimization methods, forming the basis of our sample-efficient off-policy algorithm, Policy Gradient with VRER (PG-VRER). Motivated by the lack of rigorous theoretical analysis of experience replay, we develop a novel framework that explicitly captures dependencies introduced by Markovian dynamics and behavior-policy interactions. Using this framework, we establish finite-time convergence guarantees for PG-VRER and reveal a fundamental bias-variance trade-off: reusing older experience increases bias but simultaneously reduces gradient variance. Extensive empirical experiments demonstrate that VRER consistently accelerates policy learning and improves performance over state-of-the-art policy optimization algorithms.