CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training
This addresses the challenge of data mixture optimization for language model pre-training, offering an automated solution that improves performance, though it is incremental as it builds on existing clustering and search methods.
The paper tackles the problem of identifying optimal pre-training data mixtures for language models, which are typically unlabeled and labor-intensive to curate, by proposing CLIMB, an automated framework that discovers and refines mixtures through clustering and iterative search. The result is a 1B model trained on 400B tokens that exceeds the state-of-the-art Llama-3.2-1B by 2.0%, with domain-specific optimizations yielding up to 5% improvement over random sampling.
Pre-training datasets are typically collected from web content and lack inherent domain divisions. For instance, widely used datasets like Common Crawl do not include explicit domain labels, while manually curating labeled datasets such as The Pile is labor-intensive. Consequently, identifying an optimal pre-training data mixture remains a challenging problem, despite its significant benefits for pre-training performance. To address these challenges, we propose CLustering-based Iterative Data Mixture Bootstrapping (CLIMB), an automated framework that discovers, evaluates, and refines data mixtures in a pre-training setting. Specifically, CLIMB embeds and clusters large-scale datasets in a semantic space and then iteratively searches for optimal mixtures using a smaller proxy model and a predictor. When continuously trained on 400B tokens with this mixture, our 1B model exceeds the state-of-the-art Llama-3.2-1B by 2.0%. Moreover, we observe that optimizing for a specific domain (e.g., Social Sciences) yields a 5% improvement over random sampling. Finally, we introduce ClimbLab, a filtered 1.2-trillion-token corpus with 20 clusters as a research playground, and ClimbMix, a compact yet powerful 400-billion-token dataset designed for efficient pre-training that delivers superior performance under an equal token budget. We analyze the final data mixture, elucidating the characteristics of an optimal data mixture. Our data is available at: https://research.nvidia.com/labs/lpr/climb/