Select and Improve: Understanding the Mechanics of Post-Training for Reasoning
Provides mechanistic understanding of RL post-training for reasoning, offering practical insights for scaling reasoning capabilities in language models.
The paper studies how reinforcement learning post-training enhances reasoning capabilities in models, identifying two core mechanisms: strategy selection and strategy improvement. Experiments with Qwen-2.5-1.5B show that SFT data enables strategy selection and increasing difficulty in RL data enables strategy improvement.
Reinforcement learning has rapidly emerged as a key component in the training of reasoning and coding models, yet it remains poorly understood from a mechanistic perspective. We study how and through what underlying processes capabilities are acquired or enhanced via reinforcement learning post-training. Our analysis, based on controlled math reasoning experiments with Qwen-2.5-1.5B, reveals two core mechanisms: strategy selection and strategy improvement. Our results highlight the role of SFT data and reinforcement learning data in activating these mechanisms, in particular showing how supervising the model on diverse reasoning strategies can enable strategy selection and how increasing difficulty in reinforcement learning data can enable strategy improvement. Taken together, our results provide mechanistic insight into RL training and suggest practical interventions to continue scaling reasoning capabilities.