Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweighting
This work addresses inefficiency in RLVR for improving large language models on reasoning tasks, offering a practical improvement for researchers and practitioners, though it is incremental as it builds on existing GRPO methods.
The paper tackled the problem of wasted compute in reinforcement learning with verifiable rewards (RLVR) due to negative groups yielding zero gradient, by proposing LENS, a method that assigns confidence-dependent rewards to incorrect responses to make negative groups informative. On the MATH benchmark with models like Llama-3.1-8B and Qwen-2.5-3B, LENS consistently outperformed the GRPO baseline, showing significant gains on harder items.
Reinforcement learning with verifiable rewards (RLVR) has become a standard recipe for improving large language models (LLMs) on reasoning tasks, with Group Relative Policy Optimization (GRPO) widely used in practice. Yet GRPO wastes substantial compute on negative groups: groups in which no sampled response is correct yield zero advantage and thus no gradient. We ask whether negative groups can be leveraged without extra supervision. Starting from a maximum-likelihood (MLE) objective in reward modeling, we show that the MLE gradient is equivalent to a policy gradient for a modified value function. This value function adds a confidence-weighted penalty on incorrect responses, imposing larger penalties on more confident mistakes. We refer to this as \textbf{L}ikelihood \textbf{E}stimation with \textbf{N}egative \textbf{S}amples (\textbf{LENS}). LENS modifies GRPO to assign non-zero, confidence-dependent rewards to incorrect generations, making negative groups informative and converting previously wasted samples into useful gradient updates. On the MATH benchmark with Llama-3.1-8B and Qwen-2.5-3B, the proposed variant consistently outperforms GRPO baseline, with significant gains on harder items. These results demonstrate a principled and practical way to "rescue" negative groups, improving efficiency and performance in RLVR.