A Generalist Framework for Panoptic Segmentation of Images and Videos
This provides a generalist framework for panoptic segmentation in images and videos, which is incremental as it adapts existing diffusion models to this task.
The authors tackled panoptic segmentation by formulating it as a discrete data generation problem using a diffusion model, achieving competitive performance with state-of-the-art specialist methods in similar settings.
Panoptic segmentation assigns semantic and instance ID labels to every pixel of an image. As permutations of instance IDs are also valid solutions, the task requires learning of high-dimensional one-to-many mapping. As a result, state-of-the-art approaches use customized architectures and task-specific loss functions. We formulate panoptic segmentation as a discrete data generation problem, without relying on inductive bias of the task. A diffusion model is proposed to model panoptic masks, with a simple architecture and generic loss function. By simply adding past predictions as a conditioning signal, our method is capable of modeling video (in a streaming setting) and thereby learns to track object instances automatically. With extensive experiments, we demonstrate that our simple approach can perform competitively to state-of-the-art specialist methods in similar settings.