Improving LLM Leaderboards with Psychometrical Methodology
This work addresses the need for better evaluation metrics in AI research, offering an incremental improvement for researchers and practitioners using LLM benchmarks.
The paper tackled the problem of simplistic aggregation methods in LLM leaderboards, such as averaging scores, by applying psychometric methodologies to improve model rankings, resulting in more robust and meaningful evaluations as demonstrated with Hugging Face Leaderboard data.
The rapid development of large language models (LLMs) has necessitated the creation of benchmarks to evaluate their performance. These benchmarks resemble human tests and surveys, as they consist of sets of questions designed to measure emergent properties in the cognitive behavior of these systems. However, unlike the well-defined traits and abilities studied in social sciences, the properties measured by these benchmarks are often vaguer and less rigorously defined. The most prominent benchmarks are often grouped into leaderboards for convenience, aggregating performance metrics and enabling comparisons between models. Unfortunately, these leaderboards typically rely on simplistic aggregation methods, such as taking the average score across benchmarks. In this paper, we demonstrate the advantages of applying contemporary psychometric methodologies - originally developed for human tests and surveys - to improve the ranking of large language models on leaderboards. Using data from the Hugging Face Leaderboard as an example, we compare the results of the conventional naive ranking approach with a psychometrically informed ranking. The findings highlight the benefits of adopting psychometric techniques for more robust and meaningful evaluation of LLM performance.