What Should Baby Models Read? Exploring Sample-Efficient Data Composition on Model Performance
This work addresses the problem of optimizing data selection for sample-efficient training of language models, which is incremental as it builds on existing research by exploring specific dataset impacts.
The study investigated how pre-training data composition affects small language models' performance with limited data (10 million words), finding that models like GPT2-97M and Llama-360M performed better on complex datasets like Gutenberg, while child-directed speech and simplified stories underperformed across all sizes.
We explore the impact of pre-training data composition on the performance of small language models in a sample-efficient setting. Using datasets limited to 10 million words, we evaluate several dataset sources, including child-directed speech (CHILDES), classic books (Gutenberg), synthetic data (TinyStories), and a mix of these (Mix) across different model sizes ranging from 18 million to 705 million parameters. Our experiments show that smaller models (e.g., GPT2-97M, GPT2-705M, Llama-360M) perform better when trained on more complex and rich datasets like Gutenberg. Models trained on the CHILDES and TinyStories datasets underperformed across all model sizes. These findings suggest that the optimal dataset for sample efficient training depends on the model size, and that neither child-directed speech nor simplified stories are optimal for language models of all sizes. We highlight the importance of considering both dataset composition and model capacity for effective sample efficient language model training.