On the Hidden Objective Biases of Group-based Reinforcement Learning
This work addresses fundamental limitations in widely used methods for post-training large language models, providing theoretical insights for future improvements.
The paper identifies structural mismatches between reward optimization and training objectives in group-based reinforcement learning methods like GRPO, revealing systematic gradient biases, insensitivity to reward scaling, and momentum-driven policy update issues.
Group-based reinforcement learning methods, like Group Relative Policy Optimization (GRPO), are widely used nowadays to post-train large language models. Despite their empirical success, they exhibit structural mismatches between reward optimization and the underlying training objective. In this paper, we present a theoretical analysis of GRPO style methods by studying them within a unified surrogate formulation. This perspective reveals recurring properties that affect all the methods under analysis: (i) non-uniform group weighting induces systematic gradient biases on shared prefix tokens; (ii) interactions with the AdamW optimizer make training dynamics largely insensitive to reward scaling; and (iii) optimizer momentum can push policy updates beyond the intended clipping region under repeated optimization steps. We believe that these findings highlight fundamental limitations of current approaches and provide principled guidance for the design of future formulations.