Image retrieval outperforms diffusion models on data augmentation
This work addresses the effectiveness of data augmentation methods for researchers and practitioners in computer vision, showing that incremental improvements from diffusion models may not yet surpass simpler alternatives.
The paper tackled the problem of using diffusion models for data augmentation in classification tasks, finding that a simple image retrieval method using the pre-training data outperformed diffusion-based generation, with retrieval achieving stronger downstream performance.
Many approaches have been proposed to use diffusion models to augment training datasets for downstream tasks, such as classification. However, diffusion models are themselves trained on large datasets, often with noisy annotations, and it remains an open question to which extent these models contribute to downstream classification performance. In particular, it remains unclear if they generalize enough to improve over directly using the additional data of their pre-training process for augmentation. We systematically evaluate a range of existing methods to generate images from diffusion models and study new extensions to assess their benefit for data augmentation. Personalizing diffusion models towards the target data outperforms simpler prompting strategies. However, using the pre-training data of the diffusion model alone, via a simple nearest-neighbor retrieval procedure, leads to even stronger downstream performance. Our study explores the potential of diffusion models in generating new training data, and surprisingly finds that these sophisticated models are not yet able to beat a simple and strong image retrieval baseline on simple downstream vision tasks.