An Empirical Exploration in Quality Filtering of Text Data
This work addresses a problem for machine learning practitioners by revealing that over-filtering data can harm model performance, highlighting the need for better filtering strategies, though it is incremental in nature.
The study found that aggressive filtering of low-quality text data, such as from Common Crawl, can decrease model quality on downstream tasks for GPT-like language models, contrary to conventional wisdom, with results showing performance drops across various tasks.
While conventional wisdom suggests that more aggressively filtering data from low-quality sources like Common Crawl always monotonically improves the quality of training data, we find that aggressive filtering can in fact lead to a decrease in model quality on a wide array of downstream tasks for a GPT-like language model. We speculate that this is because optimizing sufficiently strongly for a proxy metric harms performance on the true objective, suggesting a need for more robust filtering objectives when attempting to filter more aggressively. We hope this work leads to detailed analysis of the effects of dataset filtering design choices on downstream model performance in future work.