DALL-E for Detection: Language-driven Compositional Image Synthesis for Object Detection
This addresses the challenge of data scarcity for object detection tasks, offering a scalable method for generating diverse training data, but it is incremental as it builds on existing text-to-image models.
The paper tackles the problem of generating labeled training data for object detection by using text-to-image synthesis frameworks like DALL-E to create foreground object masks and background context images, which are then composited to augment datasets. The approach is demonstrated on Pascal VOC and COCO, showing advantages in out-of-distribution and zero-shot scenarios, though no specific performance numbers are provided.
We propose a new paradigm to automatically generate training data with accurate labels at scale using the text-toimage synthesis frameworks (e.g., DALL-E, Stable Diffusion, etc.). The proposed approach decouples training data generation into foreground object mask generation and background (context) image generation. For foreground object mask generation, we use a simple textual template with object class name as input to DALL-E to generate a diverse set of foreground images. A foreground-background segmentation algorithm is then used to generate foreground object masks. Next, in order to generate context images, first a language description of the context is generated by applying an image captioning method on a small set of images representing the context. These language descriptions are then used to generate diverse sets of context images using the DALL-E framework. These are then composited with object masks generated in the first step to provide an augmented training set for a classifier. We demonstrate the advantages of our approach on four object detection datasets including on Pascal VOC and COCO object detection tasks. Furthermore, we also highlight the compositional nature of our data generation approach on out-of-distribution and zero-shot data generation scenarios.