Will Large-scale Generative Models Corrupt Future Datasets?
This addresses a potential data quality problem for the computer vision research community, highlighting risks in dataset curation as generative models become widespread, though it is incremental in exploring contamination effects.
The paper investigates whether large-scale text-to-image generative models, such as DALL·E 2 and StableDiffusion, could degrade the quality of future datasets and computer vision model performance by simulating contamination with generated images. Experiments on ImageNet-scale and COCO-scale datasets show that generated images negatively affect downstream tasks like image classification and generation, with impact varying by task and contamination amount.
Recently proposed large-scale text-to-image generative models such as DALL$\cdot$E 2, Midjourney, and StableDiffusion can generate high-quality and realistic images from users' prompts. Not limited to the research community, ordinary Internet users enjoy these generative models, and consequently, a tremendous amount of generated images have been shared on the Internet. Meanwhile, today's success of deep learning in the computer vision field owes a lot to images collected from the Internet. These trends lead us to a research question: "\textbf{will such generated images impact the quality of future datasets and the performance of computer vision models positively or negatively?}" This paper empirically answers this question by simulating contamination. Namely, we generate ImageNet-scale and COCO-scale datasets using a state-of-the-art generative model and evaluate models trained with "contaminated" datasets on various tasks, including image classification and image generation. Throughout experiments, we conclude that generated images negatively affect downstream performance, while the significance depends on tasks and the amount of generated images. The generated datasets and the codes for experiments will be publicly released for future research. Generated datasets and source codes are available from \url{https://github.com/moskomule/dataset-contamination}.