SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training
This addresses efficiency issues in LLM pre-training for researchers and practitioners, offering a complementary approach to existing deduplication methods.
The paper tackles the problem of duplicated data slowing down language model pre-training by proposing a soft deduplication method that reduces sampling weights for highly common data, achieving at least a 26% reduction in training steps while maintaining comparable perplexity and improving downstream accuracy by 1.77%.
The effectiveness of large language models (LLMs) is often hindered by duplicated data in their extensive pre-training datasets. Current approaches primarily focus on detecting and removing duplicates, which risks the loss of valuable information and neglects the varying degrees of duplication. To address this, we propose a soft deduplication method that maintains dataset integrity while selectively reducing the sampling weight of data with high commonness. Central to our approach is the concept of "data commonness", a metric we introduce to quantify the degree of duplication by measuring the occurrence probabilities of samples using an n-gram model. Empirical analysis shows that this method significantly improves training efficiency, achieving comparable perplexity scores with at least a 26% reduction in required training steps. Additionally, it enhances average few-shot downstream accuracy by 1.77% when trained for an equivalent duration. Importantly, this approach consistently improves performance, even on rigorously deduplicated datasets, indicating its potential to complement existing methods and become a standard pre-training process for LLMs.