On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation
This work addresses the challenge of enhancing reasoning capabilities in large language models for AI researchers and practitioners, offering incremental improvements by refining RLVR analysis and application.
The paper tackled the problem of understanding and improving reinforcement learning with verifiable rewards (RLVR) for large language model reasoning by focusing on the direction of updates rather than their magnitude, demonstrating that this approach more effectively identifies critical reasoning updates and leads to improved reasoning accuracy through test-time extrapolation and training-time reweighting methods.
Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models. While existing analyses identify that RLVR-induced changes are sparse, they primarily focus on the \textbf{magnitude} of these updates, largely overlooking their \textbf{direction}. In this work, we argue that the direction of updates is a more critical lens for understanding RLVR's effects, which can be captured by the signed, token-level log probability difference $Î\log p$ between the base and final RLVR models. Through statistical analysis and token-replacement interventions, we demonstrate that $Î\log p$ more effectively identifies sparse, yet reasoning-critical updates than magnitude-based metrics (\eg divergence or entropy). Building on this insight, we propose two practical applications: (1) a \textit{test-time extrapolation} method that amplifies the policy along the learned $Î\log p$ direction to improve reasoning accuracy without further training; (2) a \textit{training-time reweighting} method that focuses learning on low-probability (corresponding to higher $Î\log p$) tokens, which improves reasoning performance across models and benchmarks. Our work establishes the direction of change as a key principle for analyzing and improving RLVR.