ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoning
This addresses the issue of overthinking in reasoning models, improving efficiency for users in reasoning-intensive tasks, though it is incremental as it builds on existing reinforcement learning and reasoning methods.
The paper tackles the problem of inefficient and redundant reasoning in models like OpenAI o1 and DeepSeek-R1 by proposing ShorterBetter, a reinforcement learning method that enables models to learn optimal Chain-of-Thought lengths, achieving a 50%-80% reduction in output lengths while maintaining accuracy.
Recent models such as OpenAI o1 and DeepSeek-R1 have demonstrated strong performance on reasoning-intensive tasks by generating extended Chain-of-Thought (CoT) traces. While longer reasoning helps with thorough exploration of solution paths for complex problems, it also often leads to inefficient and redundant outputs--a phenomenon commonly described as overthinking. In this paper, we propose ShorterBetter, a simple yet effective reinforcement learning method that enables reasoning models to learn their own optimal CoT lengths without manual supervision. We define the Sample Optimal Length (SOL) as the length of the shortest correct response among multiple generations, which serves as a dynamic reward signal to guide the model toward efficient reasoning. Applied to DeepSeek-Distill-Qwen-1.5B/7B as base models, ShorterBetter achieves 50%-80% reduction in output lengths in both in-domain and out-of-domain reasoning tasks while maintaining accuracy. Our reasoning trace analysis shows that ShorterBetter refines the structure of the reasoning traces by reducing unnecessary repetition, excessive self-verification, and over-exploration of alternatives.