IQ Test for LLMs: An Evaluation Framework for Uncovering Core Skills in LLMs
This provides a more holistic understanding of LLM capabilities, aiding in model selection and task design for researchers and practitioners.
The authors tackled the problem of interpreting benchmark scores for large language models (LLMs) by proposing a new evaluation paradigm using factor analysis to identify latent skills, revealing that a small set of skills largely explains performance across 60 LLMs on 44 tasks.
Current evaluations of large language models (LLMs) rely on benchmark scores, but it is difficult to interpret what these individual scores reveal about a model's overall skills. Specifically, as a community we lack understanding of how tasks relate to one another, what they measure in common, how they differ, or which ones are redundant. As a result, models are often assessed via a single score averaged across benchmarks, an approach that fails to capture the models' wholistic strengths and limitations. Here, we propose a new evaluation paradigm that uses factor analysis to identify latent skills driving performance across benchmarks. We apply this method to a comprehensive new leaderboard showcasing the performance of 60 LLMs on 44 tasks, and identify a small set of latent skills that largely explain performance. Finally, we turn these insights into practical tools that identify redundant tasks, aid in model selection, and profile models along each latent skill.