Measuring Bias of Web-filtered Text Datasets and Bias Propagation Through Training
This work addresses biases in pretraining datasets for LLMs, which can affect model fairness and performance, but it is incremental as it builds on prior bias studies in computer vision datasets.
The study measured biases in popular web-filtered text datasets for large language models (LLMs) and found that neural networks could classify which dataset a text sequence came from with high accuracy, significantly better than humans, indicating biases from filtering pipelines that persist even after rewriting and propagate through model training.
We investigate biases in pretraining datasets for large language models (LLMs) through dataset classification experiments. Building on prior work demonstrating the existence of biases in popular computer vision datasets, we analyze popular open-source pretraining datasets for LLMs derived from CommonCrawl including C4, RefinedWeb, DolmaCC, RedPajama-V2, FineWeb, and DCLM-Baseline. Despite those datasets being obtained with similar curation steps, neural networks can classify surprisingly well which dataset a single text sequence belongs to, significantly better than a human can. This indicates that small differences in filtering and processing pipelines induce fingerprints evident in formatting, vocabulary, and content distributions. Those biases remain even when the text is rewritten with LLMs. Moreover, these biases propagate through training: Random sequences generated by models trained on those datasets can be classified well by a classifier trained on the original datasets. This can be leveraged to estimate the pretraining mixture proportions of the data sources.