Quantity doesn't buy quality syntax with neural language models
This highlights a key limitation in neural language models for natural language processing, showing that current approaches are incremental and insufficient for human-like syntactic performance.
The study found that increasing network size and training data for recurrent neural networks yields minimal gains in predicting grammatical words in complex contexts, with Transformer models like GPT and BERT performing worse in some cases, indicating a need for more data-efficient architectures.
Recurrent neural networks can learn to predict upcoming words remarkably well on average; in syntactically complex contexts, however, they often assign unexpectedly high probabilities to ungrammatical words. We investigate to what extent these shortcomings can be mitigated by increasing the size of the network and the corpus on which it is trained. We find that gains from increasing network size are minimal beyond a certain point. Likewise, expanding the training corpus yields diminishing returns; we estimate that the training corpus would need to be unrealistically large for the models to match human performance. A comparison to GPT and BERT, Transformer-based models trained on billions of words, reveals that these models perform even more poorly than our LSTMs in some constructions. Our results make the case for more data efficient architectures.