MVP-SEG: Multi-View Prompt Learning for Open-Vocabulary Semantic Segmentation
This work addresses the challenge of pixel-level adaptation of CLIP for semantic segmentation, which is an incremental improvement over existing methods.
The paper tackles the problem of adapting CLIP features for open-vocabulary semantic segmentation by proposing MVP-SEG, which uses multi-view prompt learning with an orthogonal constraint loss and global prompt refining, resulting in strong generalization to unseen categories and outperforming previous methods on benchmarks.
CLIP (Contrastive Language-Image Pretraining) is well-developed for open-vocabulary zero-shot image-level recognition, while its applications in pixel-level tasks are less investigated, where most efforts directly adopt CLIP features without deliberative adaptations. In this work, we first demonstrate the necessity of image-pixel CLIP feature adaption, then provide Multi-View Prompt learning (MVP-SEG) as an effective solution to achieve image-pixel adaptation and to solve open-vocabulary semantic segmentation. Concretely, MVP-SEG deliberately learns multiple prompts trained by our Orthogonal Constraint Loss (OCLoss), by which each prompt is supervised to exploit CLIP feature on different object parts, and collaborative segmentation masks generated by all prompts promote better segmentation. Moreover, MVP-SEG introduces Global Prompt Refining (GPR) to further eliminate class-wise segmentation noise. Experiments show that the multi-view prompts learned from seen categories have strong generalization to unseen categories, and MVP-SEG+ which combines the knowledge transfer stage significantly outperforms previous methods on several benchmarks. Moreover, qualitative results justify that MVP-SEG does lead to better focus on different local parts.