Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives
This work addresses the problem of efficient data mixture optimization for researchers and practitioners in AI, but it is incremental as it primarily reviews and organizes existing methods without introducing new algorithms.
The paper surveys existing methods for determining optimal domain sampling proportions in data mixtures for training large language models under computational constraints, categorizing them into fine-grained subtypes and analyzing their relationships and trade-offs.
Training large language models with data collected from various domains can improve their performance on downstream tasks. However, given a fixed training budget, the sampling proportions of these different domains significantly impact the model's performance. How can we determine the domain weights across different data domains to train the best-performing model within constrained computational resources? In this paper, we provide a comprehensive overview of existing data mixture methods. First, we propose a fine-grained categorization of existing methods, extending beyond the previous offline and online classification. Offline methods are further grouped into heuristic-based, algorithm-based, and function fitting-based methods. For online methods, we categorize them into three groups: online min-max optimization, online mixing law, and other approaches by drawing connections with the optimization frameworks underlying offline methods. Second, we summarize the problem formulations, representative algorithms for each subtype of offline and online methods, and clarify the relationships and distinctions among them. Finally, we discuss the advantages and disadvantages of each method and highlight key challenges in the field of data mixture.