Thinking Fast and Right: Balancing Accuracy and Reasoning Length with Adaptive Rewards
This work addresses the problem of high inference costs for users of large language models in mathematical reasoning tasks, though it appears incremental as it builds on existing reward-shaping approaches.
The paper tackles the problem of reinforcement learning-trained large language models producing unnecessarily long reasoning traces, which increases inference costs and latency. The proposed adaptive reward-shaping method dynamically adjusts the trade-off between accuracy and response length, resulting in consistent and dramatic reductions in reasoning length while largely maintaining accuracy across multiple datasets.
Large language models (LLMs) have demonstrated strong reasoning abilities in mathematical tasks, often enhanced through reinforcement learning (RL). However, RL-trained models frequently produce unnecessarily long reasoning traces -- even for simple queries -- leading to increased inference costs and latency. While recent approaches attempt to control verbosity by adding length penalties to the reward function, these methods rely on fixed penalty terms that are hard to tune and cannot adapt as the model's reasoning capability evolves, limiting their effectiveness. In this work, we propose an adaptive reward-shaping method that enables LLMs to "think fast and right" -- producing concise outputs without sacrificing correctness. Our method dynamically adjusts the reward trade-off between accuracy and response length based on model performance: when accuracy is high, the length penalty increases to encourage faster length reduction; when accuracy drops, the penalty is relaxed to preserve correctness. This adaptive reward accelerates early-stage length reduction while avoiding over-compression in later stages. Experiments across multiple datasets show that our approach consistently and dramatically reduces reasoning length while largely maintaining accuracy, offering a new direction for cost-efficient adaptive reasoning in large-scale language models.