The Price of a Second Thought: On the Evaluation of Reasoning Efficiency in Large Language Models
This addresses the issue of computational waste in reasoning models for AI researchers and practitioners, offering an incremental improvement through a hybrid approach.
The paper tackles the problem of reasoning efficiency in large language models by formalizing it as a relative measure between thinking and instruct models, revealing that thinking models waste computation on easy problems but provide value on harder ones. It proposes COTHINK, a two-stage pipeline that reduces token usage by 21.1% while maintaining accuracy across benchmarks like GSM8K, MATH500, and AIME24.
Recent thinking models trained with reinforcement learning and backward-checking CoT often suffer from overthinking: they produce excessively long outputs even on simple problems, wasting computation. Existing evaluations, based on token efficiency, give an incomplete view as they neglect problem difficulty and intermediate computation costs. We formalize reasoning efficiency as a relative measure between thinking and instruct models, treating instruct models as the minimal-effort baseline. A systematic study across four thinking models and multiple benchmarks reveals two consistent patterns: (i) instruct models achieve higher efficiency overall, and (ii) problem difficulty affects efficiency, with thinking models wasting computation on easy problems but providing value on harder ones. Building on this insight, we propose COTHINK, a simple two-stage pipeline: an instruct model drafts a brief outline, and a thinking model expands it. On GSM8K, MATH500, and AIME24, COTHINK cuts token usage by 21.1% while keeping accuracy on four thinking models, and remains competitive with strong efficiency baselines.