On the De-duplication of LAION-2B
This addresses copyright and usability problems for researchers and developers using large image datasets like LAION-2B for training generative models, though it is incremental as it applies existing methods to new data.
The paper tackled the problem of duplicated images in the LAION-2B dataset, which poses copyright issues for generative models, and found that roughly 700 million images (30%) are likely duplicated using an efficient algorithmic chain.
Generative models, such as DALL-E, Midjourney, and Stable Diffusion, have societal implications that extend beyond the field of computer science. These models require large image databases like LAION-2B, which contain two billion images. At this scale, manual inspection is difficult and automated analysis is challenging. In addition, recent studies show that duplicated images pose copyright problems for models trained on LAION2B, which hinders its usability. This paper proposes an algorithmic chain that runs with modest compute, that compresses CLIP features to enable efficient duplicate detection, even for vast image volumes. Our approach demonstrates that roughly 700 million images, or about 30\%, of LAION-2B's images are likely duplicated. Our method also provides the histograms of duplication on this dataset, which we use to reveal more examples of verbatim copies by Stable Diffusion and further justify the approach. The current version of the de-duplicated set will be distributed online.