Efficient Benchmarking of AI Agents
This work addresses the problem of expensive benchmarking for AI researchers and developers, offering a more efficient method for reliable leaderboard ranking, though it is incremental as it builds on existing evaluation frameworks.
The paper tackled the high cost of evaluating AI agents on comprehensive benchmarks by investigating whether small task subsets can preserve agent rankings. They found that using a mid-range difficulty filter based on historical pass rates reduces evaluation tasks by 44-70% while maintaining high rank fidelity under distribution shifts.
Evaluating AI agents on comprehensive benchmarks is expensive because each evaluation requires interactive rollouts with tool use and multi-step reasoning. We study whether small task subsets can preserve agent rankings at substantially lower cost. Unlike static language model benchmarks, agent evaluation is subject to scaffold-driven distribution shift, since performance depends on the framework wrapping the underlying model. Across eight benchmarks, 33 agent scaffolds, and 70+ model configurations, we find that absolute score prediction degrades under this shift, while rank-order prediction remains stable. Exploiting this asymmetry, we propose a simple optimization-free protocol: evaluate new agents only on tasks with intermediate historical pass rates (30-70%). This mid-range difficulty filter, motivated by Item Response Theory, reduces the number of evaluation tasks by 44-70% while maintaining high rank fidelity under scaffold and temporal shifts. It provides more reliable rankings than random sampling, which exhibits high variance across seeds, and outperforms greedy task selection under distribution shift. These results suggest that reliable leaderboard ranking does not require full-benchmark evaluation.