JE-IRT: A Geometric Lens on LLM Abilities through Joint Embedding Item Response Theory
This provides a more interpretable and unified method for evaluating LLMs, addressing the limitations of single-score metrics for researchers and practitioners, though it is incremental in applying geometric embeddings to this domain.
The authors tackled the problem of evaluating LLMs by developing JE-IRT, a geometric framework that embeds models and questions in a shared space to reveal multidimensional abilities, with results showing that larger norms indicate harder questions and out-of-distribution behavior is explained by directional alignment.
Standard LLM evaluation practices compress diverse abilities into single scores, obscuring their inherently multidimensional nature. We present JE-IRT, a geometric item-response framework that embeds both LLMs and questions in a shared space. For question embeddings, the direction encodes semantics and the norm encodes difficulty, while correctness on each question is determined by the geometric interaction between the model and question embeddings. This geometry replaces a global ranking of LLMs with topical specialization and enables smooth variation across related questions. Building on this framework, our experimental results reveal that out-of-distribution behavior can be explained through directional alignment, and that larger norms consistently indicate harder questions. Moreover, JE-IRT naturally supports generalization: once the space is learned, new LLMs are added by fitting a single embedding. The learned space further reveals an LLM-internal taxonomy that only partially aligns with human-defined subject categories. JE-IRT thus establishes a unified and interpretable geometric lens that connects LLM abilities with the structure of questions, offering a distinctive perspective on model evaluation and generalization.