Adaptive-Boundary-Clipping GRPO: Ensuring Bounded Ratios for Stable and Generalizable Training
This work addresses stability and generalization issues in GRPO for LLM-based reinforcement learning, representing an incremental improvement over existing methods.
The paper tackles the suboptimal clipping mechanism in Group Relative Policy Optimization (GRPO) for reinforcement learning with large language models, proposing Adaptive-Boundary-Clipping GRPO (ABC-GRPO) which achieves superior performance on mathematical reasoning tasks using Qwen3 LLMs and maintains higher entropy to preserve exploration capacity.
Group Relative Policy Optimization (GRPO) has emerged as a popular algorithm for reinforcement learning with large language models (LLMs). However, upon analyzing its clipping mechanism, we argue that it is suboptimal in certain scenarios. With appropriate modifications, GRPO can be significantly enhanced to improve both flexibility and generalization. To this end, we propose Adaptive-Boundary-Clipping GRPO (ABC-GRPO), an asymmetric and adaptive refinement of the original GRPO framework. We demonstrate that ABC-GRPO achieves superior performance over standard GRPO on mathematical reasoning tasks using the Qwen3 LLMs. Moreover, ABC-GRPO maintains substantially higher entropy throughout training, thereby preserving the model's exploration capacity and mitigating premature convergence. The implementation code is available online to ease reproducibility https://github.com/chi2liu/ABC-GRPO.