Predicting Disagreement with Human Raters in LLM-as-a-Judge Difficulty Assessment without Using Generation-Time Probability Signals
For educators and researchers using LLMs to automatically assign difficulty levels to educational materials, this method reduces human effort by identifying ratings likely to be incorrect without requiring probability signals.
The paper proposes a method to predict when LLM-generated difficulty ratings will disagree with human raters, using geometric consistency in an embedding space rather than generation-time probability signals. It achieves higher AUC than probability-based baselines on English CEFR-based sentence difficulty assessment with GPT-OSS-120B and Qwen3-235B-A22B.
Automatic generation of educational materials using large language models (LLMs) is becoming increasingly common, but assigning difficulty levels to such materials still requires substantial human effort. LLM-as-a-Judge has therefore attracted attention, yet disagreement with human raters remains a major challenge. We propose a method for predicting which LLM-generated difficulty ratings are likely to disagree with human raters, so that such cases can be sent for re-rating. Unlike prior approaches, our method does not rely on generation-time probability signals, which must be collected during rating generation and are often difficult to compare across LLMs. Instead, exploiting the fact that difficulty is an ordinal scale, we use a separate embedding space, such as ModernBERT, and identify disagreement candidates based on the geometric consistency of the rating set. Experiments on English CEFR-based sentence difficulty assessment with GPT-OSS-120B and Qwen3-235B-A22B showed that the proposed method achieved higher AUC for predicting disagreement with human raters than probability-based baselines.