ConciseRL: Conciseness-Guided Reinforcement Learning for Efficient Reasoning Models
This addresses efficiency and hallucination issues in reasoning models for AI applications, though it is incremental as it builds on existing RL and LLM judge methods.
The paper tackles the problem of overly long reasoning traces in large language models by introducing a conciseness-guided reinforcement learning method, achieving up to 31x token reduction and 7% accuracy improvement on simple MATH problems, and +7.5% accuracy with 3.6x fewer tokens on hard problems.
Large language models excel at complex tasks by breaking down problems into structured reasoning steps. However, reasoning traces often extend beyond reaching a correct answer, causing wasted computation, reduced readability, and hallucinations. To address this, we introduce a novel hyperparameter-free conciseness score used as a reward signal within a reinforcement learning framework to guide models toward generating correct and concise reasoning traces. This score is evaluated by a large language model acting as a judge, enabling dynamic, context-aware feedback beyond simple token length. Our method achieves state-of-the-art efficiency-accuracy trade-offs on the MATH dataset, reducing token usage by up to 31x on simple problems while improving accuracy by 7%, and on the hardest problems, it outperforms full reasoning by +7.5% accuracy with up to 3.6x fewer tokens. On TheoremQA, our method improves accuracy by +2.2% using 12.5x fewer tokens. We also conduct ablation studies on the judge model, reward composition, and problem difficulty, showing that our method dynamically adapts reasoning length based on problem difficulty and benefits significantly from stronger judges. The code, model weights, and datasets are open-sourced at https://github.com/RazvanDu/ConciseRL.