Automated Benchmark Auditing for AI Agents and Large Language Models
For AI benchmark developers and users, this work provides an automated method to detect flaws that distort model rankings, addressing a critical reliability problem in LLM and agent evaluation.
The paper introduces Auto Benchmark Audit (ABA), an agentic framework that audits benchmark tasks for issues like hidden dependencies and incorrect ground truths. Applied to 168 benchmarks, it found issues in over 25.7% of tasks, and removing problematic tasks increased average performance on SWE-bench Verified and Terminal-Bench 2 by 9.9% and 9.6%, respectively.
Modern AI benchmarks operate at a complexity that outpaces traditional verification methods. Tasks authored by domain experts often contain implicit assumptions, incomplete environment specifications, and brittle evaluation logic that human annotation cannot reliably catch. We introduce Auto Benchmark Audit (ABA), an agentic framework that systematically audits individual benchmark tasks, uncovering issues such as hidden environment dependencies, specification gaps, and limited grading logic. We run ABA on a collection of frontier LLM benchmarks and previous NeurIPS publications, totaling 168 benchmarks across nine domains. Across this corpus, ABA identifies critical issues including ambiguous task design, execution environment conflicts, and incorrect ground truths in over 25.7% of the evaluated tasks. The precision of these automated audits is validated by expert review and independent third-party reports such as upstream PRs. Crucially, we demonstrate that these problematic tasks severely distorts capability assessments for agents and LLMs: filtering out these tasks with issues shifts model rankings and increases average performance on SWE-bench Verified and Terminal-Bench 2 by 9.9% and 9.6%, respectively. We release the agentic tool and all task annotations to support the future development of frontier benchmarks.