An Approximate Ascent Approach To Prove Convergence of PPO
This work addresses a fundamental theoretical gap for researchers and practitioners using PPO in reinforcement learning, though it is incremental as it builds on existing methods.
The paper tackled the incomplete theoretical foundations of Proximal Policy Optimization (PPO) by proving its convergence under standard assumptions and identifying an issue in truncated Generalized Advantage Estimation that causes infinite mass collapse at episode boundaries. They showed that a simple weight correction yields substantial improvements in environments like Lunar Lander.
Proximal Policy Optimization (PPO) is among the most widely used deep reinforcement learning algorithms, yet its theoretical foundations remain incomplete. Most importantly, convergence and understanding of fundamental PPO advantages remain widely open. Under standard theory assumptions we show how PPO's policy update scheme (performing multiple epochs of minibatch updates on multi-use rollouts with a surrogate gradient) can be interpreted as approximated policy gradient ascent. We show how to control the bias accumulated by the surrogate gradients and use techniques from random reshuffling to prove a convergence theorem for PPO that sheds light on PPO's success. Additionally, we identify a previously overlooked issue in truncated Generalized Advantage Estimation commonly used in PPO. The geometric weighting scheme induces infinite mass collapse onto the longest $k$-step advantage estimator at episode boundaries. Empirical evaluations show that a simple weight correction can yield substantial improvements in environments with strong terminal signal, such as Lunar Lander.