What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret
This addresses a critical bottleneck for researchers and practitioners using reinforcement learning to improve reasoning in large language models, though it is an incremental improvement focused on a specific method.
The paper tackles the failure of Proximal Policy Optimization (PPO) in long chain-of-thought tasks for large language models, proposing Value-Calibrated PPO (VC-PPO) to address value initialization bias and reward signal decay, which significantly boosts performance on the American Invitational Mathematics Examination (AIME).
Reinforcement learning (RL) is pivotal for enabling large language models (LLMs) to generate long chains of thought (CoT) for complex tasks like math and reasoning. However, Proximal Policy Optimization (PPO), effective in many RL scenarios, fails in long CoT tasks. This paper identifies that value initialization bias and reward signal decay are the root causes of PPO's failure. We propose Value-Calibrated PPO (VC-PPO) to address these issues. In VC-PPO, the value model is pretrained to tackle initialization bias, and the Generalized Advantage Estimation (GAE) computation is decoupled between the actor and critic to mitigate reward signal decay. Experiments on the American Invitational Mathematics Examination (AIME) show that VC-PPO significantly boosts PPO performance. Ablation studies show that techniques in VC-PPO are essential in enhancing PPO for long CoT tasks.