CLIP-S$^4$: Language-Guided Self-Supervised Semantic Segmentation
This work addresses the need for more flexible and annotation-free semantic segmentation methods in computer vision, offering a novel approach that reduces reliance on labeled data and predefined classes, though it builds incrementally on existing vision-language models.
The paper tackles the problem of costly pixel-wise annotations and predefined classes in semantic segmentation by introducing CLIP-S^4, which leverages self-supervised pixel representation learning and vision-language models to enable various segmentation tasks without human annotations or unknown class information, resulting in consistent and substantial performance improvements over state-of-the-art methods on four benchmarks and large margins in unknown class recognition.
Existing semantic segmentation approaches are often limited by costly pixel-wise annotations and predefined classes. In this work, we present CLIP-S$^4$ that leverages self-supervised pixel representation learning and vision-language models to enable various semantic segmentation tasks (e.g., unsupervised, transfer learning, language-driven segmentation) without any human annotations and unknown class information. We first learn pixel embeddings with pixel-segment contrastive learning from different augmented views of images. To further improve the pixel embeddings and enable language-driven semantic segmentation, we design two types of consistency guided by vision-language models: 1) embedding consistency, aligning our pixel embeddings to the joint feature space of a pre-trained vision-language model, CLIP; and 2) semantic consistency, forcing our model to make the same predictions as CLIP over a set of carefully designed target classes with both known and unknown prototypes. Thus, CLIP-S$^4$ enables a new task of class-free semantic segmentation where no unknown class information is needed during training. As a result, our approach shows consistent and substantial performance improvement over four popular benchmarks compared with the state-of-the-art unsupervised and language-driven semantic segmentation methods. More importantly, our method outperforms these methods on unknown class recognition by a large margin.