On Predictability of Reinforcement Learning Dynamics for Large Language Models
This work addresses the challenge of inefficient and opaque RL training for LLMs, offering a practical acceleration tool for large-scale applications, though it is incremental in building on existing RL methods.
The paper tackled the problem of understanding and predicting reinforcement learning (RL) dynamics in large language models (LLMs), identifying rank-1 dominance and linear dynamics that enable accurate prediction of performance gains, and proposed AlphaRL to achieve up to 2.5x speedup while retaining over 96% of reasoning performance.
Recent advances in reasoning capabilities of large language models (LLMs) are largely driven by reinforcement learning (RL), yet the underlying parameter dynamics during RL training remain poorly understood. This work identifies two fundamental properties of RL-induced parameter updates in LLMs: (1) Rank-1 Dominance, where the top singular subspace of the parameter update matrix nearly fully determines reasoning improvements, recovering over 99\% of performance gains; and (2) Rank-1 Linear Dynamics, where this dominant subspace evolves linearly throughout training, enabling accurate prediction from early checkpoints. Extensive experiments across 8 LLMs and 7 algorithms validate the generalizability of these properties. More importantly, based on these findings, we propose AlphaRL, a plug-in acceleration framework that extrapolates the final parameter update using a short early training window, achieving up to 2.5 speedup while retaining \textgreater 96\% of reasoning performance without extra modules or hyperparameter tuning. This positions our finding as a versatile and practical tool for large-scale RL, opening a path toward principled, interpretable, and efficient training paradigm for LLMs.