Prior-based Noisy Text Data Filtering: Fast and Strong Alternative For Perplexity
This provides a fast and effective alternative for data filtering in LLM pretraining, addressing efficiency and reliability issues, though it is incremental as it builds on existing filtering concepts.
The paper tackles the problem of filtering noisy text data for large language model pretraining by proposing a prior-based method that uses corpus-level term frequency statistics, achieving the highest average performance across 20 downstream benchmarks and reducing time cost by over 1000x compared to perplexity-based filtering.
As large language models (LLMs) are pretrained on massive web corpora, careful selection of data becomes essential to ensure effective and efficient learning. While perplexity (PPL)-based filtering has shown strong performance, it suffers from drawbacks: substantial time costs and inherent unreliability of the model when handling noisy or out-of-distribution samples. In this work, we propose a simple yet powerful alternative: a prior-based data filtering method that estimates token priors using corpus-level term frequency statistics, inspired by linguistic insights on word roles and lexical density. Our approach filters documents based on the mean and standard deviation of token priors, serving as a fast proxy to PPL while requiring no model inference. Despite its simplicity, the prior-based filter achieves the highest average performance across 20 downstream benchmarks, while reducing time cost by over 1000x compared to PPL-based filtering. We further demonstrate its applicability to symbolic languages such as code and math, and its dynamic adaptability to multilingual corpora without supervision