Stochastic Variance Reduction for Policy Gradient Estimation
This work addresses sample efficiency issues in reinforcement learning for robotic continuous control, representing an incremental improvement by adapting an existing optimization technique to a known bottleneck.
The paper tackles the problem of high variance in policy gradient estimates for reinforcement learning, which leads to poor sample efficiency, by applying stochastic variance reduced gradient descent (SVRG) to model-free policy gradient methods, achieving significantly better performance on Mujoco tasks compared to state-of-the-art methods like TRPO.
Recent advances in policy gradient methods and deep learning have demonstrated their applicability for complex reinforcement learning problems. However, the variance of the performance gradient estimates obtained from the simulation is often excessive, leading to poor sample efficiency. In this paper, we apply the stochastic variance reduced gradient descent (SVRG) to model-free policy gradient to significantly improve the sample-efficiency. The SVRG estimation is incorporated into a trust-region Newton conjugate gradient framework for the policy optimization. On several Mujoco tasks, our method achieves significantly better performance compared to the state-of-the-art model-free policy gradient methods in robotic continuous control such as trust region policy optimization (TRPO)