Economic Evaluation of LLMs
This provides a practical tool for practitioners in AI deployment to make informed decisions about model selection by integrating economic factors, though it is incremental as it builds on existing Pareto frontier methods.
The paper tackles the problem of comparing LLMs with different performance trade-offs by proposing an economic evaluation framework that quantifies trade-offs as a single dollar value based on use-case constraints, and finds that reasoning models offer better accuracy-cost tradeoffs when mistake costs exceed $0.01, and single large LLMs often outperform cascades at mistake costs as low as $0.1.
Practitioners often navigate LLM performance trade-offs by plotting Pareto frontiers of optimal accuracy-cost trade-offs. However, this approach offers no way to compare between LLMs with distinct strengths and weaknesses: for example, a cheap, error-prone model vs a pricey but accurate one. To address this gap, we propose economic evaluation of LLMs. Our framework quantifies the performance trade-off of an LLM as a single number based on the economic constraints of a concrete use case, all expressed in dollars: the cost of making a mistake, the cost of incremental latency, and the cost of abstaining from a query. We apply our economic evaluation framework to compare the performance of reasoning and non-reasoning models on difficult questions from the MATH benchmark, discovering that reasoning models offer better accuracy-cost tradeoffs as soon as the economic cost of a mistake exceeds \$0.01. In addition, we find that single large LLMs often outperform cascades when the cost of making a mistake is as low as \$0.1. Overall, our findings suggest that when automating meaningful human tasks with AI models, practitioners should typically use the most powerful available model, rather than attempt to minimize AI deployment costs, since deployment costs are likely dwarfed by the economic impact of AI errors.