Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
This work addresses the challenge of enhancing reasoning abilities in language models through scalable training techniques, though it appears incremental relative to existing RL approaches.
The researchers investigated how prolonged reinforcement learning training affects reasoning capabilities in small language models, achieving significant improvements of +14.7% on math, +13.9% on coding, and +54.8% on logic puzzle tasks over strong baselines.
Recent advancements in reasoning-focused language models such as OpenAI's O1 and DeepSeek-R1 have shown that scaling test-time computation-through chain-of-thought reasoning and iterative exploration-can yield substantial improvements on complex tasks like mathematics and code generation. These breakthroughs have been driven by large-scale reinforcement learning (RL), particularly when combined with verifiable reward signals that provide objective and grounded supervision. In this report, we investigate the effects of prolonged reinforcement learning on a small language model across a diverse set of reasoning domains. Our work identifies several key ingredients for effective training, including the use of verifiable reward tasks, enhancements to Group Relative Policy Optimization (GRPO), and practical techniques to improve training stability and generalization. We introduce controlled KL regularization, clipping ratio, and periodic reference policy resets as critical components for unlocking long-term performance gains. Our model achieves significant improvements over strong baselines, including +14.7% on math, +13.9% on coding, and +54.8% on logic puzzle tasks. To facilitate continued research, we release our model publicly.