Dataset Enhancement with Instance-Level Augmentations
This method addresses the need for enhanced dataset generalization and privacy anonymization in computer vision, though it is incremental as it builds on existing diffusion models.
The authors tackled the problem of limited dataset diversity by introducing instance-level data augmentation using pre-trained latent diffusion models to repaint objects in images, which improved state-of-the-art models for tasks like object detection and semantic segmentation, with performance gains reported on datasets such as COCO and Pascal VOC.
We present a method for expanding a dataset by incorporating knowledge from the wide distribution of pre-trained latent diffusion models. Data augmentations typically incorporate inductive biases about the image formation process into the training (e.g. translation, scaling, colour changes, etc.). Here, we go beyond simple pixel transformations and introduce the concept of instance-level data augmentation by repainting parts of the image at the level of object instances. The method combines a conditional diffusion model with depth and edge maps control conditioning to seamlessly repaint individual objects inside the scene, being applicable to any segmentation or detection dataset. Used as a data augmentation method, it improves the performance and generalization of the state-of-the-art salient object detection, semantic segmentation and object detection models. By redrawing all privacy-sensitive instances (people, license plates, etc.), the method is also applicable for data anonymization. We also release fully synthetic and anonymized expansions for popular datasets: COCO, Pascal VOC and DUTS.