Reliability Crisis of Reference-free Metrics for Grammatical Error Correction
This exposes a reliability crisis in automatic evaluation for GEC, which can mislead users in selecting systems, and is incremental as it builds on existing metrics.
The paper tackles the problem of adversarial systems exploiting reference-free metrics for grammatical error correction, demonstrating that their proposed adversarial attack strategies outperform the current state-of-the-art.
Reference-free evaluation metrics for grammatical error correction (GEC) have achieved high correlation with human judgments. However, these metrics are not designed to evaluate adversarial systems that aim to obtain unjustifiably high scores. The existence of such systems undermines the reliability of automatic evaluation, as it can mislead users in selecting appropriate GEC systems. In this study, we propose adversarial attack strategies for four reference-free metrics: SOME, Scribendi, IMPARA, and LLM-based metrics, and demonstrate that our adversarial systems outperform the current state-of-the-art. These findings highlight the need for more robust evaluation methods.