FreeSeg: Free Mask from Interpretable Contrastive Language-Image Pretraining for Semantic Segmentation
This work addresses the high annotation cost in semantic segmentation for open-world scenarios, offering a novel approach that eliminates the need for dense masks.
The paper tackles the problem of semantic segmentation without pixel-level annotations by using natural language supervision, achieving a 13.4% higher mIoU on the VOC dataset compared to previous state-of-the-art methods.
Fully supervised semantic segmentation learns from dense masks, which requires heavy annotation cost for closed set. In this paper, we use natural language as supervision without any pixel-level annotation for open world segmentation. We call the proposed framework as FreeSeg, where the mask is freely available from raw feature map of pretraining model. Compared with zero-shot or openset segmentation, FreeSeg doesn't require any annotated masks, and it widely predicts categories beyond class-agnostic unsupervised segmentation. Specifically, FreeSeg obtains free mask from Image-Text Similarity Map (ITSM) of Interpretable Contrastive Language-Image Pretraining (ICLIP). And our core improvements are the smoothed min pooling for dense ICLIP, with the partial label and pixel strategies for segmentation. Furthermore, FreeSeg is very straight forward without complex design like grouping, clustering or retrieval. Besides the simplicity, the performances of FreeSeg surpass previous state-of-the-art at large margins, e.g. 13.4% higher at mIoU on VOC dataset in the same settings.