What Does Your Benchmark Really Measure? A Framework for Robust Inference of AI Capabilities
This addresses the issue of unreliable AI capability assessments for researchers and practitioners, representing a foundational shift rather than an incremental improvement.
The paper tackles the problem of unreliable benchmark evaluations in AI by proposing a framework that treats evaluation as inference from a theory of capability, and as a proof of concept, it introduces methods that account for uncertainty from sensitivity and finite samples, including an adaptive algorithm that significantly reduces sample complexity.
Evaluations of generative models on benchmark data are now ubiquitous, and their outcomes critically shape public and scientific expectations of AI's capabilities. Yet growing skepticism surrounds their reliability. How can we know that a reported accuracy genuinely reflects a model's true performance? Evaluations are often presented as simple measurements, but in reality they are inferences: to treat benchmark scores as evidence of capability is already to assume a theory of what capability is and how it manifests in a test. We make this step explicit by proposing a principled framework for evaluation as inference: begin from a theory of capability, and then derive methods for estimating it. This perspective, familiar in fields such as psychometrics, has not yet become commonplace in AI evaluation. As a proof of concept, we address a central challenge that undermines reliability: sensitivity to perturbations. After formulating a model of ability, we introduce methods that infer ability while accounting for uncertainty from sensitivity and finite samples, including an adaptive algorithm that significantly reduces sample complexity. Together, these contributions lay the groundwork for more reliable and trustworthy estimates of AI capabilities as measured through benchmarks.