An Interpretable and Scalable Framework for Evaluating Large Language Models
For researchers and practitioners evaluating LLMs, this framework provides a computationally efficient and theoretically grounded alternative to standard benchmarking, enabling more principled analysis of model abilities and item characteristics.
The paper proposes a scalable and interpretable framework for evaluating large language models using item response theory, reformulated as constrained matrix factorization via majorization-minimization. It achieves orders-of-magnitude speedups over existing methods while maintaining estimation accuracy on benchmarks like MATH-500 and Open LLM Leaderboard.
Evaluation of large language models (LLMs) is increasingly critical, yet standard benchmarking methods rely on average accuracy, overlooking both the inherent stochasticity of LLM outputs and the heterogeneity of benchmark items. Item Response Theory (IRT) offers a principled framework for modeling latent model abilities and item characteristics, but conventional methods are computationally expensive and numerically unstable, limiting large-scale implementations. To address these challenges, we propose an interpretable and scalable framework for LLM evaluation based on the majorization-minimization principle. Our approach reformulates the problem as a sequence of constrained matrix factorization subproblems, enabling stable and efficient parameter estimation with theoretical guarantees for identifiability and convergence. Experiments on synthetic and real-world datasets, including MATH-500 and six Open LLM Leaderboard benchmarks, demonstrate that our method achieves superior scalability and interpretability. It delivers orders-of-magnitude speedups over competing methods while maintaining comparable or even higher estimation accuracy. Our results align with established scaling laws and offer insights into item difficulty and discrimination, informing more principled benchmark design.