Sample Complexity of Estimating the Policy Gradient for Nearly Deterministic Dynamical Systems
This work addresses the challenge of efficient policy gradient estimation for robotics control, providing a theoretical framework that could improve learning in nearly deterministic systems, though it is incremental as it builds on existing finite-difference methods.
The paper tackles the problem of high variance in policy gradient estimates for reinforcement learning in robotics by analyzing nearly deterministic dynamical systems, showing that finite-difference methods can have substantially lower variance than traditional theorem-based approaches, with empirical validation on an inverted pendulum task.
Reinforcement learning is a promising approach to learning robotics controllers. It has recently been shown that algorithms based on finite-difference estimates of the policy gradient are competitive with algorithms based on the policy gradient theorem. We propose a theoretical framework for understanding this phenomenon. Our key insight is that many dynamical systems (especially those of interest in robotics control tasks) are nearly deterministic -- i.e., they can be modeled as a deterministic system with a small stochastic perturbation. We show that for such systems, finite-difference estimates of the policy gradient can have substantially lower variance than estimates based on the policy gradient theorem. Finally, we empirically evaluate our insights in an experiment on the inverted pendulum.