Filling in the Mechanisms: How do LMs Learn Filler-Gap Dependencies under Developmental Constraints?
For researchers in language acquisition and NLP, this work highlights the data efficiency gap between humans and LMs, emphasizing the need for inductive biases in models.
The study investigates whether language models (LMs) trained on developmentally feasible data learn shared representations for filler-gap dependencies across different syntactic constructions. Results suggest shared but item-sensitive mechanisms emerge with limited data, yet LMs require far more data than humans, indicating a need for language-specific biases.
For humans, filler-gap dependencies require a shared representation across different syntactic constructions. Although causal analyses suggest this may also be true for LLMs (Boguraev et al., 2025), it is still unclear if such a representation also exists for language models trained on developmentally feasible quantities of data. We applied Distributed Alignment Search (DAS, Geiger et al. (2024)) to LMs trained on varying amounts of data from the BabyLM challenge (Warstadt et al., 2023), to evaluate whether representations of filler-gap dependencies transfer between wh-questions and topicalization, which greatly vary in terms of their input frequency. Our results suggest shared, yet item-sensitive mechanisms may develop with limited training data. More importantly, LMs still require far more data than humans to learn comparable generalizations, highlighting the need for language-specific biases in models of language acquisition.