A Training-Free Framework for Open-Vocabulary Image Segmentation and Recognition with EfficientNet and CLIP
This addresses the problem of segmenting and recognizing objects from arbitrary categories in images without requiring task-specific training, which is useful for computer vision applications, though it is incremental as it builds on existing models like CLIP.
The paper tackles open-vocabulary image segmentation and recognition by proposing a training-free framework that combines EfficientNetB0 for unsupervised segmentation and CLIP for recognition, achieving state-of-the-art performance on benchmarks like COCO, ADE20K, and PASCAL VOC with metrics such as Hungarian mIoU, precision, recall, and F1-score.
This paper presents a novel training-free framework for open-vocabulary image segmentation and object recognition (OVSR), which leverages EfficientNetB0, a convolutional neural network, for unsupervised segmentation and CLIP, a vision-language model, for open-vocabulary object recognition. The proposed framework adopts a two stage pipeline: unsupervised image segmentation followed by segment-level recognition via vision-language alignment. In the first stage, pixel-wise features extracted from EfficientNetB0 are decomposed using singular value decomposition to obtain latent representations, which are then clustered using hierarchical clustering to segment semantically meaningful regions. The number of clusters is adaptively determined by the distribution of singular values. In the second stage, the segmented regions are localized and encoded into image embeddings using the Vision Transformer backbone of CLIP. Text embeddings are precomputed using CLIP's text encoder from category-specific prompts, including a generic something else prompt to support open set recognition. The image and text embeddings are concatenated and projected into a shared latent feature space via SVD to enhance cross-modal alignment. Recognition is performed by computing the softmax over the similarities between the projected image and text embeddings. The proposed method is evaluated on standard benchmarks, including COCO, ADE20K, and PASCAL VOC, achieving state-of-the-art performance in terms of Hungarian mIoU, precision, recall, and F1-score. These results demonstrate the effectiveness, flexibility, and generalizability of the proposed framework.