Deprecating Benchmarks: Criteria and Framework
This addresses a critical issue for AI researchers, developers, and policymakers to prevent misleading assessments of model capabilities.
The paper tackles the problem of outdated benchmarks in AI by proposing criteria and a framework for deprecating them, aiming to improve evaluation rigor and quality for frontier models.
As frontier artificial intelligence (AI) models rapidly advance, benchmarks are integral to comparing different models and measuring their progress in different task-specific domains. However, there is a lack of guidance on when and how benchmarks should be deprecated once they cease to effectively perform their purpose. This risks benchmark scores over-valuing model capabilities, or worse, obscuring capabilities and safety-washing. Based on a review of benchmarking practices, we propose criteria to decide when to fully or partially deprecate benchmarks, and a framework for deprecating benchmarks. Our work aims to advance the state of benchmarking towards rigorous and quality evaluations, especially for frontier models, and our recommendations are aimed to benefit benchmark developers, benchmark users, AI governance actors (across governments, academia, and industry panels), and policy makers.