Automatic Metric Validation for Grammatical Error Correction
This addresses a methodological bottleneck for researchers and practitioners in GEC by providing a more efficient validation approach, though it appears incremental as it builds on existing validation practices.
The paper tackles the problem of costly and unreliable metric validation in Grammatical Error Correction (GEC) by proposing MAEGE, an automatic methodology that reveals issues like the poor corpus-level ranking performance of the standard M^2 metric.
Metric validation in Grammatical Error Correction (GEC) is currently done by observing the correlation between human and metric-induced rankings. However, such correlation studies are costly, methodologically troublesome, and suffer from low inter-rater agreement. We propose MAEGE, an automatic methodology for GEC metric validation, that overcomes many of the difficulties with existing practices. Experiments with \maege\ shed a new light on metric quality, showing for example that the standard $M^2$ metric fares poorly on corpus-level ranking. Moreover, we use MAEGE to perform a detailed analysis of metric behavior, showing that correcting some types of errors is consistently penalized by existing metrics.