Learning from Child-Directed Speech in Two-Language Scenarios: A French-English Case Study
This work addresses the gap in research on developmentally plausible language models for multilingual scenarios, specifically French-English, but is incremental as it extends existing frameworks and methods.
The study tackled the problem of developing compact language models in multilingual settings by extending BabyBERTa to English-French scenarios under size-matched data conditions, finding that training on Wikipedia benefits semantic tasks while child-directed speech improves grammatical judgments in monolingual settings, with bilingual pretraining yielding notable gains for textual entailment, particularly strong improvements for French.
Research on developmentally plausible language models has largely focused on English, leaving open questions about multilingual settings. We present a systematic study of compact language models by extending BabyBERTa to English-French scenarios under strictly size-matched data conditions, covering monolingual, bilingual, and cross-lingual settings. Our design contrasts two types of training corpora: (i) child-directed speech (about 2.5M tokens), following BabyBERTa and related work, and (ii) multi-domain corpora (about 10M tokens), extending the BabyLM framework to French. To enable fair evaluation, we also introduce new resources, including French versions of QAMR and QASRL, as well as English and French multi-domain corpora. We evaluate the models on both syntactic and semantic tasks and compare them with models trained on Wikipedia-only data. The results reveal context-dependent effects: training on Wikipedia consistently benefits semantic tasks, whereas child-directed speech improves grammatical judgments in monolingual settings. Bilingual pretraining yields notable gains for textual entailment, with particularly strong improvements for French. Importantly, similar patterns emerge across BabyBERTa, RoBERTa, and LTG-BERT, suggesting consistent trends across architectures.