Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory
This work provides a systematic method for diagnosing the reliability of LLM-as-a-Judge, which is crucial for researchers and practitioners relying on automated evaluation in NLP.
This paper introduces a two-phase diagnostic framework based on Item Response Theory (IRT) to assess the reliability of LLM-as-a-Judge. The framework uses the Graded Response Model (GRM) to formalize reliability in terms of intrinsic consistency (stability under prompt variations) and human alignment (correspondence with human assessments), providing interpretable signals for diagnosing judgments.
While LLM-as-a-Judge is widely used in automated evaluation, existing validation practices primarily operate at the level of observed outputs, offering limited insight into whether LLM judges themselves function as stable and reliable measurement instruments. To address this limitation, we introduce a two-phase diagnostic framework for assessing reliability of LLM-as-a-Judge, grounded in Item Response Theory (IRT). The framework adopts Graded Response Model (GRM) of IRT and formalizes reliability along two complementary dimensions: (1) intrinsic consistency, defined as the stability of measurement behavior under prompt variations, and (2) human alignment, capturing correspondence with human quality assessments. We empirically examine diverse LLM judges with this framework, and show that leveraging IRT-GRM yields interpretable signals for diagnosing judgments systematically. These signals provide practical guidance for verifying reliablity of LLM-as-a-Judge and identifying potential causes of unreliability.