Is Child-Directed Speech Effective Training Data for Language Models?
This research addresses the problem of data efficiency in language modeling for AI researchers, showing incremental insights by challenging assumptions about training data quality.
The study investigated whether child-directed speech is effective training data for language models by training GPT-2 and RoBERTa on 29M words of such data and comparing it to other datasets, finding that local discourse properties affect performance but global developmental ordering does not, and child language input is not uniquely valuable, suggesting children's learning algorithms are more data-efficient.
While high-performing language models are typically trained on hundreds of billions of words, human children become fluent language users with a much smaller amount of data. What are the features of the data they receive, and how do these features support language modeling objectives? To investigate this question, we train GPT-2 and RoBERTa models on 29M words of English child-directed speech and a new matched, synthetic dataset (TinyDialogues), comparing to OpenSubtitles, Wikipedia, and a heterogeneous blend of datasets from the BabyLM challenge. We evaluate the syntactic and semantic knowledge of these models using developmentally-inspired evaluations. Through pretraining experiments, we test whether the global developmental ordering or the local discourse ordering of children's training data supports high performance relative to other datasets. The local properties of the data affect model results, but surprisingly, global properties do not. Further, child language input is not uniquely valuable for training language models. These findings support the hypothesis that, rather than proceeding from better data, the child's learning algorithm is substantially more data-efficient than current language modeling techniques.