Reinforcement Learning is all You Need
This demonstrates the potential of RL-only training for enhancing reasoning in language models, though it is incremental as it builds on prior RL successes like DeepSeek R1.
The authors trained a 3B language model using pure reinforcement learning on the Countdown Game, finding it outperformed baselines on four of five benchmarks and showed improved generalization beyond training data, though emergent insights did not always yield correct answers.
Inspired by the success of DeepSeek R1 in reasoning via reinforcement learning without human feedback, we train a 3B language model using the Countdown Game with pure reinforcement learning. Our model outperforms baselines on four of five benchmarks, demonstrating improved generalization beyond its training data. Notably, response length does not correlate with reasoning quality, and while "aha moments" emerge, they do not always yield correct answers. These findings highlight the potential of RL-only training for reasoning enhancement and suggest future work on refining reward structures to bridge emergent insights with accuracy.