Resolving Inconsistent Semantics in Multi-Dataset Image Segmentation
This addresses a practical problem for researchers and practitioners scaling up segmentation models with multiple datasets, though it appears incremental as it builds on existing multi-dataset training approaches.
The paper tackles the problem of semantic inconsistencies when merging multiple datasets for image segmentation training, introducing a method that integrates language-based embeddings and label space-specific queries to achieve performance improvements of 1.6% mIoU, 9.1% PQ, 12.1% AP, and 3.0% PIQ over previous methods on four benchmark datasets.
Leveraging multiple training datasets to scale up image segmentation models is beneficial for increasing robustness and semantic understanding. Individual datasets have well-defined ground truth with non-overlapping mask layouts and mutually exclusive semantics. However, merging them for multi-dataset training disrupts this harmony and leads to semantic inconsistencies; for example, the class "person" in one dataset and class "face" in another will require multilabel handling for certain pixels. Existing methods struggle with this setting, particularly when evaluated on label spaces mixed from the individual training sets. To overcome these issues, we introduce a simple yet effective multi-dataset training approach by integrating language-based embeddings of class names and label space-specific query embeddings. Our method maintains high performance regardless of the underlying inconsistencies between training datasets. Notably, on four benchmark datasets with label space inconsistencies during inference, we outperform previous methods by 1.6% mIoU for semantic segmentation, 9.1% PQ for panoptic segmentation, 12.1% AP for instance segmentation, and 3.0% in the newly proposed PIQ metric.