Agreement Between Large Language Models and Human Raters in Essay Scoring: A Research Synthesis
This research addresses the reliability of LLMs for educators and researchers in automated essay scoring, but it is incremental as it synthesizes existing findings without new empirical results.
The paper synthesized 65 studies on large language models (LLMs) in automatic essay scoring, finding that LLM-human agreement was generally moderate to good, with indices ranging from 0.30 to 0.80, but variability was high due to study-specific factors and lack of standardization.
Despite the growing promise of large language models (LLMs) in automatic essay scoring (AES), empirical findings regarding their reliability compared to human raters remain mixed. Following the PRISMA 2020 guidelines, we synthesized 65 published and unpublished studies from January 2022 to August 2025 that examined agreement between LLMs and human raters in AES. Across studies, reported LLM-human agreement was generally moderate to good, with agreement indices (e.g., Quadratic Weighted Kappa, Pearson correlation, and Spearman's rho) mostly ranging between 0.30 and 0.80. Substantial variability in agreement levels was observed across studies, reflecting differences in study-specific factors as well as the lack of standardized reporting practices. Implications and directions for future research are discussed.