Associating Spatially-Consistent Grouping with Text-supervised Semantic Segmentation
This work addresses the limitation of text-supervised semantic segmentation for computer vision applications, offering a novel hybrid approach that significantly improves performance over existing methods.
The paper tackles the problem of coarse and spurious region grouping in text-supervised semantic segmentation by associating self-supervised spatially-consistent grouping with region-level recognition, achieving 59.2% mIoU on Pascal VOC and 32.4% mIoU on Pascal Context benchmarks.
In this work, we investigate performing semantic segmentation solely through the training on image-sentence pairs. Due to the lack of dense annotations, existing text-supervised methods can only learn to group an image into semantic regions via pixel-insensitive feedback. As a result, their grouped results are coarse and often contain small spurious regions, limiting the upper-bound performance of segmentation. On the other hand, we observe that grouped results from self-supervised models are more semantically consistent and break the bottleneck of existing methods. Motivated by this, we introduce associate self-supervised spatially-consistent grouping with text-supervised semantic segmentation. Considering the part-like grouped results, we further adapt a text-supervised model from image-level to region-level recognition with two core designs. First, we encourage fine-grained alignment with a one-way noun-to-region contrastive loss, which reduces the mismatched noun-region pairs. Second, we adopt a contextually aware masking strategy to enable simultaneous recognition of all grouped regions. Coupled with spatially-consistent grouping and region-adapted recognition, our method achieves 59.2% mIoU and 32.4% mIoU on Pascal VOC and Pascal Context benchmarks, significantly surpassing the state-of-the-art methods.