JPPF: Multi-task Fusion for Consistent Panoptic-Part Segmentation
This work addresses the need for consistent multi-granularity scene understanding in computer vision, representing an incremental improvement through effective fusion of existing segmentation tasks.
The paper tackled the problem of part-aware panoptic segmentation by proposing JPPF, a parameter-free fusion method that dynamically balances inputs from semantic, instance, and part segmentations, achieving improved performance on datasets like Cityscapes Panoptic Parts and Pascal Panoptic Parts as measured by PartPQ and PWQ metrics.
Part-aware panoptic segmentation is a problem of computer vision that aims to provide a semantic understanding of the scene at multiple levels of granularity. More precisely, semantic areas, object instances, and semantic parts are predicted simultaneously. In this paper, we present our Joint Panoptic Part Fusion (JPPF) that combines the three individual segmentations effectively to obtain a panoptic-part segmentation. Two aspects are of utmost importance for this: First, a unified model for the three problems is desired that allows for mutually improved and consistent representation learning. Second, balancing the combination so that it gives equal importance to all individual results during fusion. Our proposed JPPF is parameter-free and dynamically balances its input. The method is evaluated and compared on the Cityscapes Panoptic Parts (CPP) and Pascal Panoptic Parts (PPP) datasets in terms of PartPQ and Part-Whole Quality (PWQ). In extensive experiments, we verify the importance of our fair fusion, highlight its most significant impact for areas that can be further segmented into parts, and demonstrate the generalization capabilities of our design without fine-tuning on 5 additional datasets.