L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning
This work addresses the challenge of allocating test-time compute efficiently for reasoning models, enabling fine-grained control over computational cost and accuracy, though it is incremental as it builds on existing reinforcement learning and reasoning model techniques.
The paper tackles the problem of uncontrollable reasoning length in language models by introducing Length Controlled Policy Optimization (LCPO), which trains models to adhere to user-specified length constraints while optimizing accuracy, resulting in a model that outperforms state-of-the-art methods and achieves significant performance gains, such as a 1.5B model surpassing GPT-4o at equal reasoning lengths.
Reasoning language models have shown an uncanny ability to improve performance at test-time by ``thinking longer''-that is, by generating longer chain-of-thought sequences and hence using more compute. However, the length of their chain-of-thought reasoning is not controllable, making it impossible to allocate test-time compute to achieve a desired level of performance. We introduce Length Controlled Policy Optimization (LCPO), a simple reinforcement learning method that optimizes for accuracy and adherence to user-specified length constraints. We use LCPO to train L1, a reasoning language model that produces outputs satisfying a length constraint given in its prompt. L1's length control allows for smoothly trading off computational cost and accuracy on a wide range of tasks, and outperforms the state-of-the-art S1 method for length control. Furthermore, we uncover an unexpected short chain-of-thought capability in models trained with LCPO. Specifically, using LCPO we derive Short Reasoning Models (SRMs), that exhibit similar reasoning patterns as full-length reasoning models, but can generate CoT lengths comparable to non-reasoning models. They demonstrate significant performance gains, for instance, our 1.5B L1 model surpasses GPT-4o at equal reasoning lengths. Overall, LCPO enables precise control over reasoning length, allowing for fine-grained allocation of test-time compute and accuracy. We release code and models at https://www.cmu-l3.github.io/l1