On the Learning Dynamics of RLVR at the Edge of Competence
This addresses the challenge of scaling reasoning in AI models, offering insights for designing data mixtures to improve learning efficiency, though it is incremental as it builds on existing RLVR frameworks.
The paper investigates how reinforcement learning with verifiable rewards (RLVR) overcomes long-horizon reasoning barriers by analyzing training dynamics on compositional tasks, finding that smooth difficulty spectra enable steady improvement while abrupt ones cause grokking-like plateaus.
Reinforcement learning with verifiable rewards (RLVR) has been a main driver of recent breakthroughs in large reasoning models. Yet it remains a mystery how rewards based solely on final outcomes can help overcome the long-horizon barrier to extended reasoning. To understand this, we develop a theory of the training dynamics of RL for transformers on compositional reasoning tasks. Our theory characterizes how the effectiveness of RLVR is governed by the smoothness of the difficulty spectrum. When data contains abrupt discontinuities in difficulty, learning undergoes grokking-type phase transitions, producing prolonged plateaus before progress recurs. In contrast, a smooth difficulty spectrum leads to a relay effect: persistent gradient signals on easier problems elevate the model's capabilities to the point where harder ones become tractable, resulting in steady and continuous improvement. Our theory explains how RLVR can improve performance at the edge of competence, and suggests that appropriately designed data mixtures can yield scalable gains. As a technical contribution, our analysis develops and adapts tools from Fourier analysis on finite groups to our setting. We validate the predicted mechanisms empirically via synthetic experiments.