LSHBloom: Memory-efficient, Extreme-scale Document Deduplication
This work addresses the high computational costs of deduplication for assembling large-scale training datasets, which is crucial for reducing training expenses and preventing memorization in LLMs, though it is incremental as it builds on existing MinhashLSH methods.
The paper tackles the problem of expensive runtime and memory usage in document-level deduplication for large language model training datasets by proposing LSHBloom, which replaces LSHIndex with Bloom filters. It achieves similar deduplication performance with only a marginal increase in false positives (as low as 1e-5), competitive runtime (270% faster than MinhashLSH on peS2o), and uses just 0.6% of the disk space, scaling to billions of documents with a 250% speedup and 54x space advantage.
Deduplication is a major focus for assembling and curating training datasets for large language models (LLM) -- detecting and eliminating additional instances of the same content -- in large collections of technical documents. Unrestrained, duplicates in the training dataset increase training costs and lead to undesirable properties such as memorization in trained models or cheating on evaluation. Contemporary approaches to document-level deduplication are often extremely expensive in both runtime and memory. We propose LSHBloom, an extension to MinhashLSH, which replaces the expensive LSHIndex with lightweight Bloom filters. LSHBloom demonstrates the same deduplication performance as MinhashLSH with only a marginal increase in false positives (as low as 1e-5 in our experiments); demonstrates competitive runtime (270\% faster than MinhashLSH on peS2o); and, crucially, uses just 0.6\% of the disk space required by MinhashLSH to deduplicate peS2o. We demonstrate that this space advantage scales with increased dataset size -- at the extreme scale of several billion documents, LSHBloom promises a 250\% speedup and a 54$\times$ space advantage over traditional MinHashLSH scaling deduplication of text datasets to many billions of documents.