TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraints
For researchers and engineers deploying LLMs as autonomous agents, this work identifies a critical but overlooked capability—prospective resource allocation—that is currently lacking in state-of-the-art models.
The paper introduces TRIAGE, a framework to evaluate whether LLMs can prospectively allocate limited compute resources across tasks. Results show that current models exhibit significant gaps in this metacognitive ability, revealing a previously unmeasured capability dimension.
Deploying language models as autonomous agents requires more than per-task accuracy: when an agent faces a queue of problems under a finite token budget, it must decide which to attempt, in what order, and how much compute to commit to each, all before any execution feedback is available. This is the prospective form of metacognitive control studied for decades in human cognition, yet whether language models possess it remains untested. We introduce TRIAGE, an evaluation framework in which a model receives a task pool and a token budget calibrated to its own baseline cost, and commits to a single ordered plan that jointly encodes selection, sequencing, and per-problem allocation. Plans are scored against an oracle with full knowledge of the model's solvability and cost on each problem, yielding a triage efficiency ratio on a common scale. We evaluate frontier and open-source models, with and without reasoning enabled, across competition mathematics, graduate-level science, code generation, and expert multidisciplinary knowledge, and find that current language models exhibit substantial gaps in prospective metacognitive control, revealing a previously unmeasured capability dimension with direct implications for resource-efficient agent deployment.