Multi-Dimensional Evaluation of LLMs for Grammatical Error Correction
For educators and developers of GEC systems, this work provides guidance on selecting LLM-based assistants and highlights the limitations of reference-based evaluation metrics.
This study evaluates latest-generation LLMs on grammatical error correction across edit precision, fluency, and meaning retention, finding that fine-tuned GPT-4o achieves state-of-the-art performance. It also shows that reference-based metrics underestimate GEC performance, with 73.76% of GPT-4o corrections being equally valid or superior to gold standards.
Automated assistants for Grammatical Error Correction are now embedded in educational platforms serving millions of learners, yet three critical gaps remain in this domain: (1) latest-generation Large Language Models (LLMs) lack comprehensive evaluation on grammar correction tasks; (2) whether combining these LLMs improves correction quality is unexplored; and (3) the extent to which reference-based metrics underestimate GEC system performance has not been adequately quantified. In this study, first, we evaluate latest-generation LLMs on edit precision, fluency preservation, and meaning retention, showing fine-tuned GPT-4o achieves state-of-the-art performance across all three dimensions. Second, through grammatical error type analysis we demonstrate that individual LLMs exhibit highly similar error correction patterns ($ρ=0.947$). Third, we show that reference-based metrics underestimate GEC performance with 73.76% of GPT-4o corrections different from gold standards being equally valid or even superior. These GEC evaluation findings equip educators with guidance for selecting GEC assistants that enhance rather than constrain student linguistic development. We make our data, code, and models publicly available.