A Simple Latent Diffusion Approach for Panoptic Segmentation and Mask Inpainting
This work addresses the problem of simplifying segmentation pipelines for computer vision researchers, though it is incremental as it builds upon existing diffusion models.
The paper tackles the complexity of panoptic and instance segmentation by proposing a latent diffusion approach based on Stable Diffusion, which simplifies the architecture and training process. The method achieves strong segmentation results on COCO and ADE20k datasets and demonstrates adaptability to multi-tasking with learnable task embeddings.
Panoptic and instance segmentation networks are often trained with specialized object detection modules, complex loss functions, and ad-hoc post-processing steps to manage the permutation-invariance of the instance masks. This work builds upon Stable Diffusion and proposes a latent diffusion approach for panoptic segmentation, resulting in a simple architecture that omits these complexities. Our training consists of two steps: (1) training a shallow autoencoder to project the segmentation masks to latent space; (2) training a diffusion model to allow image-conditioned sampling in latent space. This generative approach unlocks the exploration of mask completion or inpainting. The experimental validation on COCO and ADE20k yields strong segmentation results. Finally, we demonstrate our model's adaptability to multi-tasking by introducing learnable task embeddings.