MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary Instance Segmentation
This addresses the challenge of data scarcity in instance segmentation for rare and novel object categories, offering a practical solution for researchers and practitioners in computer vision.
The paper tackles the problem of generating synthetic labeled data for large vocabulary instance segmentation, particularly for rare and novel categories, by introducing MosaicFusion, a training-free diffusion-based data augmentation method that improves model performance on benchmarks like LVIS.
We present MosaicFusion, a simple yet effective diffusion-based data augmentation approach for large vocabulary instance segmentation. Our method is training-free and does not rely on any label supervision. Two key designs enable us to employ an off-the-shelf text-to-image diffusion model as a useful dataset generator for object instances and mask annotations. First, we divide an image canvas into several regions and perform a single round of diffusion process to generate multiple instances simultaneously, conditioning on different text prompts. Second, we obtain corresponding instance masks by aggregating cross-attention maps associated with object prompts across layers and diffusion time steps, followed by simple thresholding and edge-aware refinement processing. Without bells and whistles, our MosaicFusion can produce a significant amount of synthetic labeled data for both rare and novel categories. Experimental results on the challenging LVIS long-tailed and open-vocabulary benchmarks demonstrate that MosaicFusion can significantly improve the performance of existing instance segmentation models, especially for rare and novel categories. Code: https://github.com/Jiahao000/MosaicFusion.