Language Models Can Learn Exceptions to Syntactic Rules
This addresses the problem of understanding how neural networks handle linguistic exceptions for researchers in computational linguistics and cognitive science, though it is incremental in exploring existing models.
The study investigated whether language models can learn exceptions to syntactic rules, specifically English passivization restrictions, and found that GPT-2's probability distribution correlates highly with human acceptability judgments, supporting the entrenchment hypothesis with distributional evidence.
Artificial neural networks can generalize productively to novel contexts. Can they also learn exceptions to those productive rules? We explore this question using the case of restrictions on English passivization (e.g., the fact that "The vacation lasted five days" is grammatical, but "*Five days was lasted by the vacation" is not). We collect human acceptability judgments for passive sentences with a range of verbs, and show that the probability distribution defined by GPT-2, a language model, matches the human judgments with high correlation. We also show that the relative acceptability of a verb in the active vs. passive voice is positively correlated with the relative frequency of its occurrence in those voices. These results provide preliminary support for the entrenchment hypothesis, according to which learners track and uses the distributional properties of their input to learn negative exceptions to rules. At the same time, this hypothesis fails to explain the magnitude of unpassivizability demonstrated by certain individual verbs, suggesting that other cues to exceptionality are available in the linguistic input.