ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws
This addresses bias and diversity issues in data filtering for LLM pre-training, offering a novel approach that is incremental but impactful for model training.
The paper tackles the problem of bias in data quality filtering for large language models by proposing ScalingFilter, which eliminates the need for a reference dataset and instead uses perplexity differences between two models. The method improves zero-shot performance in downstream tasks and achieves an optimal balance with semantic diversity.
High-quality data is crucial for the pre-training performance of large language models. Unfortunately, existing quality filtering methods rely on a known high-quality dataset as reference, which can introduce potential bias and compromise diversity. In this paper, we propose ScalingFilter, a novel approach that evaluates text quality based on the perplexity difference between two language models trained on the same data, thereby eliminating the influence of the reference dataset in the filtering process. An theoretical analysis shows that ScalingFilter is equivalent to an inverse utilization of scaling laws. Through training models with 1.3B parameters on the same data source processed by various quality filters, we find ScalingFilter can improve zero-shot performance of pre-trained models in downstream tasks. To assess the bias introduced by quality filtering, we introduce semantic diversity, a metric of utilizing text embedding models for semantic representations. Extensive experiments reveal that semantic diversity is a reliable indicator of dataset diversity, and ScalingFilter achieves an optimal balance between downstream performance and semantic diversity.