AraMix: Recycling, Refiltering, and Deduplicating to Deliver the Largest Arabic Pretraining Corpus
This work addresses data inefficiency for researchers and practitioners in lower-resource languages like Arabic, offering a curated corpus to improve pretraining without additional web scraping, though it is incremental as it builds on existing datasets.
The authors tackled the problem of data redundancy in Arabic pretraining by combining and curating seven existing datasets, resulting in AraMix, a deduplicated corpus with approximately 178 billion tokens across 179 million documents, where nearly 60% of tokens were found to be duplicates.
We present AraMix, a deduplicated Arabic pretraining corpus containing approximately 178 billion tokens across 179 million documents. Rather than scraping the web again, AraMix demonstrates that substantial value lies in systematically reusing and curating existing pretraining datasets: we combine seven publicly available Arabic web datasets, apply quality filtering designed specifically for Arabic text to re-filter some datasets, and perform cross-dataset deduplication, both MinHash and sentence-level. This approach reveals that nearly 60% of tokens across these independently collected corpora are duplicates, redundancy that any new scraping efforts will reproduce. Our work suggests that for lower resource languages, investment in curation pipelines for existing data yields greater returns than additional web crawls, an approach that allowed us to curate the largest heavily filtered publicly available Arabic pretraining corpus.