Falling Through the Gaps: Neural Architectures as Models of Morphological Rule Learning
This work addresses the problem of morphological gaps in language modeling for linguists and AI researchers, but it is incremental as it highlights limitations in existing neural architectures without introducing new methods.
The study evaluated Transformer and RNN models on morphological rule learning across English, German, and Russian, focusing on morphological gaps where expected inflections are missing, and found that both architectures incorrectly produce inflections for these gaps, with success in English and German driven by majority forms.
Recent advances in neural architectures have revived the problem of morphological rule learning. We evaluate the Transformer as a model of morphological rule learning and compare it with Recurrent Neural Networks (RNN) on English, German, and Russian. We bring to the fore a hitherto overlooked problem, the morphological gaps, where the expected inflection of a word is missing. For example, 63 Russian verbs lack a first-person-singular present form such that one cannot comfortably say "*oščušču" ("I feel"). Even English has gaps, such as the past participle of "stride": the function of morphological inflection can be partial. Both neural architectures produce inflections that ought to be missing. Analyses reveal that Transformers recapitulate the statistical distribution of inflections in the training data, similar to RNNs. Models' success on English and German is driven by the fact that rules in these languages can be identified with the majority forms, which is not universal.