Colorless green recurrent networks dream hierarchically
This addresses the problem of understanding the linguistic capabilities of RNNs for researchers in computational linguistics and AI, showing they can acquire deeper grammatical competence beyond shallow patterns.
The study investigated whether recurrent neural networks (RNNs) trained for language modeling can learn abstract hierarchical syntactic structure, specifically predicting long-distance number agreement in four languages, and found they make reliable predictions and perform close to human levels.
Recurrent neural networks (RNNs) have achieved impressive results in a variety of linguistic processing tasks, suggesting that they can induce non-trivial properties of language. We investigate here to what extent RNNs learn to track abstract hierarchical syntactic structure. We test whether RNNs trained with a generic language modeling objective in four languages (Italian, English, Hebrew, Russian) can predict long-distance number agreement in various constructions. We include in our evaluation nonsensical sentences where RNNs cannot rely on semantic or lexical cues ("The colorless green ideas I ate with the chair sleep furiously"), and, for Italian, we compare model performance to human intuitions. Our language-model-trained RNNs make reliable predictions about long-distance agreement, and do not lag much behind human performance. We thus bring support to the hypothesis that RNNs are not just shallow-pattern extractors, but they also acquire deeper grammatical competence.