EcoAgent-Bench: Evaluating Economic Decision-Making in Budget-Constrained LLM Agents
This benchmark provides a new evaluation framework for LLM agents, highlighting a previously overlooked aspect of agent performance that is important for real-world deployment.
The paper introduces EcoAgent-Bench, a benchmark for evaluating LLM agents' economic decision-making under budget constraints, with 304 tasks and four decision types. Results show that current agents perform poorly, with tool-API agents achieving only 3.9-24.0% micro strict success and at most 7.3% economic consistency, indicating that budget-aware action selection is a distinct capability from task completion.
Agent benchmarks usually measure task completion and treat resource use as an auxiliary statistic. In deployment, however, the choice among a local lookup, broad search, composite research tool, stronger model, or human escalation is part of the task itself. We introduce EcoAgent-Bench, in which every task specifies priced actions and an explicit budget. Its 304 real-derived tasks span five families adapted from GAIA, HotpotQA, and MuSiQue, and test four decisions: avoiding unnecessary escalation, escalating when local evidence is insufficient, selecting a model tier, and stopping on unsupported premises. We evaluate seven LLM agents in tool-API and workspace-CLI settings, together with four oracle scripted controls. Micro-averaged accuracy rewards one-sided policies: always-escalate controls achieve high micro success while failing save-oriented tasks. We therefore also report an economic-consistency score (the worse of accuracy on upgrade-oriented and save-oriented family groups) which exposes this failure. Tool-API agents attain only 3.9-24.0% micro strict success (at most 7.3% economic consistency), often either stopping before warranted escalation or overspending on cheap tasks. A threshold-crossing budget sweep changes GPT-5.4's escalation rate from 0% to only 3%. These results show that completion under a budget and economical action selection are distinct properties. We release the task bundle, transformation pipeline, frozen evaluation environments, and integrity-bound result artifacts needed to study both.