DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction in the Era of Large Language Models
This addresses the reliability issue in GEC evaluation for researchers and practitioners, but it is incremental as it builds on existing metrics with a new weighting mechanism.
The paper tackles the problem of evaluating Grammatical Error Correction (GEC) models, which is challenging due to discrepancies between LLM-based corrections and gold references, by proposing DSGram, a novel evaluation framework that integrates semantic coherence, edit level, and fluency with dynamic weighting, and experimental results show it enhances evaluation effectiveness.
Evaluating the performance of Grammatical Error Correction (GEC) models has become increasingly challenging, as large language model (LLM)-based GEC systems often produce corrections that diverge from provided gold references. This discrepancy undermines the reliability of traditional reference-based evaluation metrics. In this study, we propose a novel evaluation framework for GEC models, DSGram, integrating Semantic Coherence, Edit Level, and Fluency, and utilizing a dynamic weighting mechanism. Our framework employs the Analytic Hierarchy Process (AHP) in conjunction with large language models to ascertain the relative importance of various evaluation criteria. Additionally, we develop a dataset incorporating human annotations and LLM-simulated sentences to validate our algorithms and fine-tune more cost-effective models. Experimental results indicate that our proposed approach enhances the effectiveness of GEC model evaluations.