Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels
This work tackles the problem of open-vocabulary semantic segmentation for computer vision researchers, offering a method to leverage powerful vision-language models without requiring extensive pixel-level semantic labels.
This paper addresses the challenge of adapting vision-language models like CLIP for pixel-level understanding in semantic segmentation without semantic labels. The proposed method, PixelCLIP, uses unlabeled images and masks from vision foundation models (SAM, DINO) along with an online clustering algorithm to learn general semantic concepts. PixelCLIP significantly improves upon CLIP and achieves competitive results against caption-supervised methods in open-vocabulary semantic segmentation.
Large-scale vision-language models like CLIP have demonstrated impressive open-vocabulary capabilities for image-level tasks, excelling in recognizing what objects are present. However, they struggle with pixel-level recognition tasks like semantic segmentation, which additionally require understanding where the objects are located. In this work, we propose a novel method, PixelCLIP, to adapt the CLIP image encoder for pixel-level understanding by guiding the model on where, which is achieved using unlabeled images and masks generated from vision foundation models such as SAM and DINO. To address the challenges of leveraging masks without semantic labels, we devise an online clustering algorithm using learnable class names to acquire general semantic concepts. PixelCLIP shows significant performance improvements over CLIP and competitive results compared to caption-supervised methods in open-vocabulary semantic segmentation. Project page is available at https://cvlab-kaist.github.io/PixelCLIP