Benchmarks as Microscopes: A Call for Model Metrology
This addresses the challenge for AI developers and researchers in accurately evaluating language models, but it is incremental as it builds on existing benchmarking critiques without introducing a new method.
The paper tackles the problem of assessing language model capabilities, arguing that static benchmarks are insufficient for predicting deployment performance and proposing a new discipline called model metrology to develop dynamic benchmarks. The result is a call for community-driven efforts to create better measurement tools, though no concrete numbers are provided.
Modern language models (LMs) pose a new challenge in capability assessment. Static benchmarks inevitably saturate without providing confidence in the deployment tolerances of LM-based systems, but developers nonetheless claim that their models have generalized traits such as reasoning or open-domain language understanding based on these flawed metrics. The science and practice of LMs requires a new approach to benchmarking which measures specific capabilities with dynamic assessments. To be confident in our metrics, we need a new discipline of model metrology -- one which focuses on how to generate benchmarks that predict performance under deployment. Motivated by our evaluation criteria, we outline how building a community of model metrology practitioners -- one focused on building tools and studying how to measure system capabilities -- is the best way to meet these needs to and add clarity to the AI discussion.