Unpacking Let Alone: Human-Scale Models Generalize to a Rare Construction in Form but not Meaning
This addresses the problem of understanding generalization limits in language models for rare constructions, which is incremental as it builds on prior work on human-scale data.
The study tested human-scale transformer language models on their ability to generalize to the rare English LET-ALONE construction, finding they are sensitive to its form but not its meaning, with results showing an asymmetry in sample efficiency compared to humans.
Humans have a remarkable ability to acquire and understand grammatical phenomena that are seen rarely, if ever, during childhood. Recent evidence suggests that language models with human-scale pretraining data may possess a similar ability by generalizing from frequent to rare constructions. However, it remains an open question how widespread this generalization ability is, and to what extent this knowledge extends to meanings of rare constructions, as opposed to just their forms. We fill this gap by testing human-scale transformer language models on their knowledge of both the form and meaning of the (rare and quirky) English LET-ALONE construction. To evaluate our LMs we construct a bespoke synthetic benchmark that targets syntactic and semantic properties of the construction. We find that human-scale LMs are sensitive to form, even when related constructions are filtered from the dataset. However, human-scale LMs do not make correct generalizations about LET-ALONE's meaning. These results point to an asymmetry in the current architectures' sample efficiency between language form and meaning, something which is not present in human language learners.