$λ$-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferences
This work addresses a specific bottleneck in RLHF/RLVR methods for improving LLM reasoning, offering an incremental but practical enhancement to existing frameworks.
The paper tackles the length bias problem in Group Relative Policy Optimization (GRPO) for reinforcement learning with verifiable rewards, where uniform token-level advantage assignment disadvantages shorter responses. The result is $\lambda$-GRPO, a method that introduces a learnable parameter to adaptively weight tokens, achieving average accuracy improvements of +1.9%, +1.0%, and +1.7% over GRPO on Qwen2.5 models with 1.5B, 3B, and 7B parameters, respectively.
Reinforcement Learning with Human Feedback (RLHF) has been the dominant approach for improving the reasoning capabilities of Large Language Models (LLMs). Recently, Reinforcement Learning with Verifiable Rewards (RLVR) has simplified this paradigm by replacing the reward and value models with rule-based verifiers. A prominent example is Group Relative Policy Optimization (GRPO). However, GRPO inherently suffers from a length bias, since the same advantage is uniformly assigned to all tokens of a response. As a result, longer responses distribute the reward over more tokens and thus contribute disproportionately to gradient updates. Several variants, such as DAPO and Dr. GRPO, modify the token-level aggregation of the loss, yet these methods remain heuristic and offer limited interpretability regarding their implicit token preferences. In this work, we explore the possibility of allowing the model to learn its own token preference during optimization. We unify existing frameworks under a single formulation and introduce a learnable parameter $λ$ that adaptively controls token-level weighting. We use $λ$-GRPO to denote our method, and we find that $λ$-GRPO achieves consistent improvements over vanilla GRPO and DAPO on multiple mathematical reasoning benchmarks. On Qwen2.5 models with 1.5B, 3B, and 7B parameters, $λ$-GRPO improves average accuracy by $+1.9\%$, $+1.0\%$, and $+1.7\%$ compared to GRPO, respectively. Importantly, these gains come without any modifications to the training data or additional computational cost, highlighting the effectiveness and practicality of learning token preferences.