Thinking Hard, Not Smart: Reasoning Models Fail to Ration Test-Time Compute Across Questions
This work identifies a new challenge for reasoning models in resource-constrained, multi-question scenarios, which is relevant for anyone deploying these models under real-world computational limits.
This paper introduces an exam-style evaluation framework where language models must distribute a shared token budget across multiple questions of varying difficulty and point values to maximize a total score. They found that current reasoning models fail to strategically allocate this budget, instead acting as greedy sequential solvers that prioritize questions by presentation order and front-load effort, showing insensitivity to question value or difficulty.
Reasoning language models increasingly use test-time compute to improve performance, but existing evaluations typically study this compute one question at a time. Yet when multiple problems share an end-to-end cost or latency constraint, models must decide how to divide limited inference compute among them. We introduce an exam-style evaluation framework for studying this setting, in which a model must distribute one shared token budget across questions with different difficulty and point values to maximize its total score. Across several open and frontier reasoning models, we find that models fail to allocate a shared budget strategically across questions of varying difficulties and values. Models behave largely as greedy sequential solvers: they prioritize questions by presentation order, front-load effort on early questions, and remain insensitive to value, with these tendencies becoming more pronounced as the number of questions grows. Explicit planning prompts spread compute more evenly but do not produce value- or difficulty-aware prioritization. The same behavioral pattern extends from mathematical to code reasoning. These findings establish global budget allocation as a distinct capability that is not captured by conventional per-question evaluation and remains a challenge for current reasoning models.