ToReMi: Topic-Aware Data Reweighting for Dynamic Pre-Training Data Selection
This work addresses the problem of efficient data selection for LLM pre-training, offering a novel method that enhances model training with domain-specific improvements, though it is incremental in advancing existing data selection techniques.
The paper tackles the challenge of selecting pre-training data for large language models by introducing ToReMi, a framework that dynamically adjusts sample weights based on topical associations and learning patterns, resulting in accelerated perplexity reduction and improved performance on downstream tasks.
Pre-training large language models (LLMs) necessitates enormous diverse textual corpora, making effective data selection a key challenge for balancing computational resources and model performance. Current methodologies primarily emphasize data quality metrics and mixing proportions, yet they fail to adequately capture the underlying semantic connections between training samples and quality disparities within individual domains. We introduce ToReMi (Topic-based Reweighting for Model improvement), a novel two-stage framework that dynamically adjusts training sample weights according to their topical associations and observed learning patterns. Our comprehensive experiments reveal that ToReMi variants consistently achieve superior performance over conventional pre-training approaches, demonstrating accelerated perplexity reduction across multiple domains and enhanced capabilities on downstream evaluation tasks. Code is available at https://github.com/zxx000728/ToReMi.