Do Grammatical Error Correction Models Realize Grammatical Generalization?
This addresses the problem of data inefficiency and generalization failure in GEC models for real-world deployment, highlighting an incremental limitation in current approaches.
The study investigated whether grammatical error correction (GEC) models can generalize grammatical knowledge to correct unseen errors, finding that a standard Transformer-based model fails to achieve this even in simple settings with limited vocabulary and syntax.
There has been an increased interest in data generation approaches to grammatical error correction (GEC) using pseudo data. However, these approaches suffer from several issues that make them inconvenient for real-world deployment including a demand for large amounts of training data. On the other hand, some errors based on grammatical rules may not necessarily require a large amount of data if GEC models can realize grammatical generalization. This study explores to what extent GEC models generalize grammatical knowledge required for correcting errors. We introduce an analysis method using synthetic and real GEC datasets with controlled vocabularies to evaluate whether models can generalize to unseen errors. We found that a current standard Transformer-based GEC model fails to realize grammatical generalization even in simple settings with limited vocabulary and syntax, suggesting that it lacks the generalization ability required to correct errors from provided training examples.