Humans and transformer LMs: Abstraction drives language learning
This research provides insights into language acquisition models by demonstrating abstraction's role in LM learning, though it is incremental as it applies existing methods to analyze known phenomena.
The study investigated how transformer language models learn linguistic categories by comparing their training behavior to human language acquisition theories, finding that abstract class-level behavior emerges earlier than item-specific behavior and that different linguistic behaviors appear abruptly at distinct training stages.
Categorization is a core component of human linguistic competence. We investigate how a transformer-based language model (LM) learns linguistic categories by comparing its behaviour over the course of training to behaviours which characterize abstract feature-based and concrete exemplar-based accounts of human language acquisition. We investigate how lexical semantic and syntactic categories emerge using novel divergence-based metrics that track learning trajectories using next-token distributions. In experiments with GPT-2 small, we find that (i) when a construction is learned, abstract class-level behaviour is evident at earlier steps than lexical item-specific behaviour, and (ii) that different linguistic behaviours emerge abruptly in sequence at different points in training, revealing that abstraction plays a key role in how LMs learn. This result informs the models of human language acquisition that LMs may serve as an existence proof for.