Improving Panoptic Segmentation at All Scales
This work provides significant improvements in panoptic segmentation performance for researchers and practitioners working with high-resolution images, particularly for large objects that are often problematic with crop-based methods.
This paper addresses the issue of large object truncation and omission in crop-based training for panoptic segmentation on multi-megapixel images. By introducing a novel crop-aware bounding box regression loss and a new data sampling/augmentation strategy, the authors achieve state-of-the-art results on Mapillary Vistas (+4.5% PQ, +5.2% mAP), Indian Driving, and Cityscapes datasets.
Crop-based training strategies decouple training resolution from GPU memory consumption, allowing the use of large-capacity panoptic segmentation networks on multi-megapixel images. Using crops, however, can introduce a bias towards truncating or missing large objects. To address this, we propose a novel crop-aware bounding box regression loss (CABB loss), which promotes predictions to be consistent with the visible parts of the cropped objects, while not over-penalizing them for extending outside of the crop. We further introduce a novel data sampling and augmentation strategy which improves generalization across scales by counteracting the imbalanced distribution of object sizes. Combining these two contributions with a carefully designed, top-down panoptic segmentation architecture, we obtain new state-of-the-art results on the challenging Mapillary Vistas (MVD), Indian Driving and Cityscapes datasets, surpassing the previously best approach on MVD by +4.5% PQ and +5.2% mAP.