Class Enhancement Losses with Pseudo Labels for Zero-shot Semantic Segmentation
This work improves zero-shot semantic segmentation for computer vision applications, offering a flexible solution that can also be applied to open-vocabulary segmentation, though it is incremental in nature.
The paper tackles the problem of zero-shot semantic segmentation by addressing over-learning to background embeddings and ignoring semantic relationships, proposing class enhancement losses and a pseudo label generation pipeline, resulting in overall best performance on benchmark datasets.
Recent mask proposal models have significantly improved the performance of zero-shot semantic segmentation. However, the use of a `background' embedding during training in these methods is problematic as the resulting model tends to over-learn and assign all unseen classes as the background class instead of their correct labels. Furthermore, they ignore the semantic relationship of text embeddings, which arguably can be highly informative for zero-shot prediction as seen classes may have close relationship with unseen classes. To this end, this paper proposes novel class enhancement losses to bypass the use of the background embbedding during training, and simultaneously exploit the semantic relationship between text embeddings and mask proposals by ranking the similarity scores. To further capture the relationship between seen and unseen classes, we propose an effective pseudo label generation pipeline using pretrained vision-language model. Extensive experiments on several benchmark datasets show that our method achieves overall the best performance for zero-shot semantic segmentation. Our method is flexible, and can also be applied to the challenging open-vocabulary semantic segmentation problem.