Stabilizing Policy Gradient Methods via Reward Profiling
This addresses the issue of unstable and slow convergence in policy gradient methods for reinforcement learning practitioners, offering a general solution that is theoretically grounded and empirically validated, though it is incremental as it builds on existing methods.
The paper tackles the problem of high variance and unreliable reward improvements in policy gradient methods for reinforcement learning by proposing a universal reward profiling framework that selectively updates policies based on high-confidence performance estimations. The result includes up to 1.5x faster convergence and up to 1.75x reduction in return variance on continuous-control benchmarks.
Policy gradient methods, which have been extensively studied in the last decade, offer an effective and efficient framework for reinforcement learning problems. However, their performances can often be unsatisfactory, suffering from unreliable reward improvements and slow convergence, due to high variance in gradient estimations. In this paper, we propose a universal reward profiling framework that can be seamlessly integrated with any policy gradient algorithm, where we selectively update the policy based on high-confidence performance estimations. We theoretically justify that our technique will not slow down the convergence of the baseline policy gradient methods, but with high probability, will result in stable and monotonic improvements of their performance. Empirically, on eight continuous-control benchmarks (Box2D and MuJoCo/PyBullet), our profiling yields up to 1.5x faster convergence to near-optimal returns, up to 1.75x reduction in return variance on some setups. Our profiling approach offers a general, theoretically grounded path to more reliable and efficient policy learning in complex environments.