How LLMs Learn to Reason: A Complex Network Perspective
This provides a new physical intuition for engineering reasoning capabilities in AI systems, though it appears incremental as it builds on existing RLVR methods.
The paper tackles the puzzling behaviors in training large language models with Reinforcement Learning from Verifiable Rewards (RLVR), such as two-stage learning curves and catastrophic forgetting, by proposing a unifying theory that models reasoning as the self-organization of a sparse semantic complex network. The result is Annealed-RLVR, an algorithm that outperforms standard RLVR on benchmarks with a 1.5B-parameter model.
Training large language models with Reinforcement Learning from Verifiable Rewards (RLVR) exhibits a set of distinctive and puzzling behaviors that remain poorly understood, including a two-stage learning curve, V-shaped response-length trajectories, and a pronounced vulnerability to catastrophic forgetting. In this work, we propose that these seemingly disparate phenomena can be explained using a single unifying theory: the model's reasoning process maps to the self-organization of a semantic complex network whose topology remains persistently sparse, with the average degree pinned close to two. This topology imposes a fundamental mechanism for forgetting and learning: it first drives the system into a maximally frustrated state where ``skill islands'' form, slow-learning happens, and forgetting is induced; then it enters a sharp growth phase where the new skills are ``bolted on'', driven by phase-transition-like learning at the web's frontier. Equipped with the theory, we propose \textit{Annealed-RLVR}, a principled algorithm that introduces an SFT-based ``heating'' step at the point of maximal frustration to resolve the competitive bottleneck and enhance the reasoning capability of the model. Experiments on a 1.5B-parameter model demonstrate that the approach outperforms standard RLVR on both in-distribution and out-of-distribution benchmarks. By recasting RLVR from black-box optimization into a predictable process of structural self-organization, our work provides a new physical intuition for engineering the emergent reasoning capabilities of future AI systems.