Rewards as Labels: Revisiting RLVR from a Classification Perspective
This work improves the training stability and performance of Large Language Models in complex mathematical reasoning tasks, particularly for researchers and practitioners using RLVR methods.
This paper addresses issues of Gradient Misassignment in Positives and Gradient Domination in Negatives in existing Reinforcement Learning with Verifiable Rewards (RLVR) methods. The authors propose Rewards as Labels (REAL), a novel framework that reformulates policy optimization as a classification problem by treating verifiable rewards as categorical labels. REAL improves average Pass@1 over DAPO by 6.7% on a 1.5B model and by 6.2% on a 7B model on mathematical reasoning benchmarks.
Reinforcement Learning with Verifiable Rewards has recently advanced the capabilities of Large Language Models in complex reasoning tasks by providing explicit rule-based supervision. Among RLVR methods, GRPO and its variants have achieved strong empirical performance. Despite their success, we identify that they suffer from Gradient Misassignment in Positives and Gradient Domination in Negatives, which lead to inefficient and suboptimal policy updates. To address these issues, we propose Rewards as Labels (REAL), a novel framework that revisits verifiable rewards as categorical labels rather than scalar weights, thereby reformulating policy optimization as a classification problem. Building on this, we further introduce anchor logits to enhance policy learning. Our analysis reveals that REAL induces a monotonic and bounded gradient weighting, enabling balanced gradient allocation across rollouts and effectively mitigating the identified mismatches. Extensive experiments on mathematical reasoning benchmarks show that REAL improves training stability and consistently outperforms GRPO and strong variants such as DAPO. On the 1.5B model, REAL improves average Pass@1 over DAPO by 6.7%. These gains further scale to 7B model, REAL continues to outperform DAPO and GSPO by 6.2% and 1.7%, respectively. Notably, even with a vanilla binary cross-entropy, REAL remains stable and exceeds DAPO by 4.5% on average.