Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theory
For researchers and practitioners in automated short answer grading, this work provides a more nuanced evaluation method that goes beyond aggregate metrics to expose model robustness differences and failure patterns.
The authors introduce an item response theory (IRT) framework to evaluate LLM-based automated short answer grading, revealing that models with similar aggregate performance differ in how accuracy declines with response difficulty, and that errors on difficult responses often involve intermediate-label collapse. They analyze semantic and linguistic correlates of difficulty, finding associations with weaker semantic alignment, stronger contradiction signals, and greater semantic isolation.
Automated short answer grading (ASAG) with large language models (LLMs) is commonly evaluated with aggregate metrics such as macro-F1 and Cohen's kappa. However, these metrics provide limited insight into how grading performance varies across student responses of differing grading difficulty. We introduce an evaluation framework for LLM-based ASAG based on item response theory (IRT), which models grading correctness as a function of latent grader ability and response grading difficulty. This formulation enables response-level analysis of where LLM graders succeed or fail and reveals robustness differences that are not visible from aggregate scores alone. We apply the framework to 17 open-weight LLMs on the SciEntsBank and Beetle benchmarks. The results show that even models with similar overall performance differ substantially in how sharply their grading accuracy declines as response difficulty increases. In addition, confusion patterns show that errors on difficult responses concentrate disproportionately on the \texttt{partially\_correct\_incomplete} label, indicating a tendency toward intermediate-label collapse under ambiguity. To characterize difficult responses, we further analyze semantic and linguistic correlates of estimated difficulty. Across both datasets, higher difficulty is associated with weaker semantic alignment to the reference answer, stronger contradiction signals, and greater semantic isolation in embedding space. Overall, these results show that item response theory offers a useful framework for evaluating LLM-based ASAG beyond aggregate performance measures.