Annotation Free Semantic Segmentation with Vision Foundation Models
This work addresses the high annotation cost in semantic segmentation for computer vision researchers, offering an incremental improvement by combining existing foundation models in a novel way.
The paper tackles the problem of expensive pixel-level annotations in semantic segmentation by generating free annotations using existing foundation models (CLIP and SAM) and aligning patch features with a pretrained text encoder for zero-shot segmentation, achieving competitive performance with minimal training and no manual annotations.
Semantic Segmentation is one of the most challenging vision tasks, usually requiring large amounts of training data with expensive pixel level annotations. With the success of foundation models and especially vision-language models, recent works attempt to achieve zeroshot semantic segmentation while requiring either large-scale training or additional image/pixel level annotations. In this work, we generate free annotations for any semantic segmentation dataset using existing foundation models. We use CLIP to detect objects and SAM to generate high quality object masks. Next, we build a lightweight module on top of a self-supervised vision encoder, DinoV2, to align the patch features with a pretrained text encoder for zeroshot semantic segmentation. Our approach can bring language-based semantics to any pretrained vision encoder with minimal training, uses foundation models as the sole source of supervision and generalizes from little training data with no annotation.