CLIP-DIY: CLIP Dense Inference Yields Open-Vocabulary Semantic Segmentation For-Free
This work addresses the problem of dense image segmentation in open-world settings for computer vision researchers, offering a training-free solution that is incremental by building on existing CLIP and localization methods.
The authors tackled open-vocabulary semantic segmentation by proposing CLIP-DIY, a method that leverages CLIP's classification abilities on multi-scale patches and unsupervised object localization without additional training, achieving state-of-the-art zero-shot results on PASCAL VOC and competitive performance on COCO.
The emergence of CLIP has opened the way for open-world image perception. The zero-shot classification capabilities of the model are impressive but are harder to use for dense tasks such as image segmentation. Several methods have proposed different modifications and learning schemes to produce dense output. Instead, we propose in this work an open-vocabulary semantic segmentation method, dubbed CLIP-DIY, which does not require any additional training or annotations, but instead leverages existing unsupervised object localization approaches. In particular, CLIP-DIY is a multi-scale approach that directly exploits CLIP classification abilities on patches of different sizes and aggregates the decision in a single map. We further guide the segmentation using foreground/background scores obtained using unsupervised object localization methods. With our method, we obtain state-of-the-art zero-shot semantic segmentation results on PASCAL VOC and perform on par with the best methods on COCO. The code is available at http://github.com/wysoczanska/clip-diy