Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Quality
This work addresses the practical constraint of data scarcity for researchers and practitioners scaling large language models, though it is incremental as it builds on existing data filtering techniques.
The study tackled the problem of limited data volume in large language model training by investigating the effects of repeating aggressively filtered datasets and manipulating document counts, finding that repeating datasets for up to ten epochs can outperform training on a ten times larger superset across multiple compute budgets.
Data filtering has become a powerful tool for improving model performance while reducing computational cost. However, as large language model compute budgets continue to grow, the limited data volume provided by heavily filtered and deduplicated datasets will become a practical constraint. In efforts to better understand how to proceed, we study model performance at various compute budgets and across multiple pre-training datasets created through data filtering and deduplication. We find that, given appropriate modifications to the training recipe, repeating existing aggressively filtered datasets for up to ten epochs can outperform training on the ten times larger superset for a single epoch across multiple compute budget orders of magnitude. While this finding relies on repeating the dataset for many epochs, we also investigate repeats within these datasets at the document level. We find that not all documents within a dataset are equal, and we can create better datasets relative to a token budget by explicitly manipulating the counts of individual documents. We conclude by arguing that even as large language models scale, data filtering remains an important direction of research.