Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
It tackles the problem of unreliable evaluations for researchers in natural language processing, offering incremental improvements through statistical techniques.
The paper addresses the lack of statistical rigor in language model evaluations by proposing methods to analyze evaluation data, measure differences between models, and plan experiments, recommending practices to minimize statistical noise and maximize informativeness.
Evaluations are critical for understanding the capabilities of large language models (LLMs). Fundamentally, evaluations are experiments; but the literature on evaluations has largely ignored the literature from other sciences on experiment analysis and planning. This article shows researchers with some training in statistics how to think about and analyze data from language model evaluations. Conceptualizing evaluation questions as having been drawn from an unseen super-population, we present formulas for analyzing evaluation data, measuring differences between two models, and planning an evaluation experiment. We make a number of specific recommendations for running language model evaluations and reporting experiment results in a way that minimizes statistical noise and maximizes informativeness.