Walk Before You Run! Concise LLM Reasoning via Reinforcement Learning
This addresses efficiency and clarity issues in LLM reasoning for users of advanced AI systems, though it appears incremental as it builds on existing reinforcement learning methods.
The paper tackles the problem of overthinking in LLMs, where long Chain-of-Thought responses show excessive redundancy, by proposing ConciseR, a two-stage reinforcement learning framework that enforces conciseness while maintaining accuracy. The model outperforms state-of-the-art reasoning models across multiple benchmarks including AIME 2024, MATH-500, and AMC 2023.
As test-time scaling becomes a pivotal research frontier in Large Language Models (LLMs) development, contemporary and advanced post-training methodologies increasingly focus on extending the generation length of long Chain-of-Thought (CoT) responses to enhance reasoning capabilities toward DeepSeek R1-like performance. However, recent studies reveal a persistent overthinking phenomenon in state-of-the-art reasoning models, manifesting as excessive redundancy or repetitive thinking patterns in long CoT responses. To address this issue, in this paper, we propose a simple yet effective two-stage reinforcement learning framework for achieving concise reasoning in LLMs, named ConciseR. Specifically, the first stage, using more training steps, aims to incentivize the model's reasoning capabilities via Group Relative Policy Optimization with clip-higher and dynamic sampling components (GRPO++), and the second stage, using fewer training steps, explicitly enforces conciseness and improves efficiency via Length-aware Group Relative Policy Optimization (L-GRPO). Significantly, ConciseR only optimizes response length once all rollouts of a sample are correct, following the "walk before you run" principle. Extensive experimental results demonstrate that our ConciseR model, which generates more concise CoT reasoning responses, outperforms recent state-of-the-art reasoning models with zero RL paradigm across AIME 2024, MATH-500, AMC 2023, Minerva, and Olympiad benchmarks.