ScribbleGen: Generative Data Augmentation Improves Scribble-supervised Semantic Segmentation
This work addresses the challenge of limited labeled data in semantic segmentation for computer vision applications, offering a novel approach to enhance scribble-based supervision, though it is incremental in leveraging existing generative models.
The paper tackles the problem of scribble-supervised semantic segmentation by proposing ScribbleGen, a generative data augmentation method using a ControlNet diffusion model, which reduces the performance gap with fully-supervised segmentation and improves results on small datasets, even surpassing fully-supervised methods in some cases.
Recent advances in generative models, such as diffusion models, have made generating high-quality synthetic images widely accessible. Prior works have shown that training on synthetic images improves many perception tasks, such as image classification, object detection, and semantic segmentation. We are the first to explore generative data augmentations for scribble-supervised semantic segmentation. We propose ScribbleGen, a generative data augmentation method that leverages a ControlNet diffusion model conditioned on semantic scribbles to produce high-quality training data. However, naive implementations of generative data augmentations may inadvertently harm the performance of the downstream segmentor rather than improve it. We leverage classifier-free diffusion guidance to enforce class consistency and introduce encode ratios to trade off data diversity for data realism. Using the guidance scale and encode ratio, we can generate a spectrum of high-quality training images. We propose multiple augmentation schemes and find that these schemes significantly impact model performance, especially in the low-data regime. Our framework further reduces the gap between the performance of scribble-supervised segmentation and that of fully-supervised segmentation. We also show that our framework significantly improves segmentation performance on small datasets, even surpassing fully-supervised segmentation. The code is available at https://github.com/mengtang-lab/scribblegen.