A Simple and Generalist Approach for Panoptic Segmentation
This addresses the need for simpler, more general solutions in computer vision for tasks like panoptic segmentation, though it is incremental as it builds on existing pretrained models and focuses on a specific domain.
The paper tackles panoptic segmentation by proposing a simple generalist framework using a deep encoder-shallow decoder architecture with per-pixel prediction, achieving a panoptic quality (PQ) of 55.1 on the MS-COCO dataset, which is state-of-the-art among generalist methods.
Panoptic segmentation is an important computer vision task, where the current state-of-the-art solutions require specialized components to perform well. We propose a simple generalist framework based on a deep encoder - shallow decoder architecture with per-pixel prediction. Essentially fine-tuning a massively pretrained image model with minimal additional components. Naively this method does not yield good results. We show that this is due to imbalance during training and propose a novel method for reducing it - centroid regression in the space of spectral positional embeddings. Our method achieves panoptic quality (PQ) of 55.1 on the challenging MS-COCO dataset, state-of-the-art performance among generalist methods.