Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning
This addresses a critical issue in reinforcement learning for LLMs, where length inflation can degrade efficiency and quality, offering a general and lossless approach that is incremental but improves upon prior methods.
The paper tackles the problem of length inflation in reinforcement learning for LLMs, where models become verbose to maximize rewards, and presents Group Relative Reward Rescaling (GR^3) as a solution that reframes length control as multiplicative rescaling, achieving significant mitigation of length inflation while maintaining comparable training dynamics and downstream performance to standard methods.
Reinforcement learning significantly enhances LLM capabilities but suffers from a critical issue: length inflation, where models adopt verbosity or inefficient reasoning to maximize rewards. Prior approaches struggle to address this challenge in a general and lossless manner, primarily because additive penalties introduce a compensatory effect that creates optimization shortcuts, while heuristic gating strategies lack generality beyond binary feedback. To bridge this gap, we present Group Relative Reward Rescaling (GR$^3$), which reframes length control as a multiplicative rescaling paradigm, effectively establishing a generalized, continuous, and reward-dependent gating mechanism. To further ensure lossless optimization, we incorporate group-relative regularization and advantage-aware calibration, which dynamically adapt length budgets to instance difficulty and preserve the advantage signal of high-quality trajectories. Empirically, across both RLHF and RLVR settings, GR$^3$~maintains training dynamics and downstream performance comparable to standard GRPO while significantly mitigating length inflation, outperforming state-of-the-art length-regularized baselines.