Zero-Shot Semantic Segmentation via Spatial and Multi-Scale Aware Visual Class Embedding
This work addresses the problem of expensive labeling costs in semantic segmentation for computer vision researchers, offering an incremental improvement by enhancing generalization in zero-shot settings.
The paper tackles the generalization limitation in language-model-based zero-shot semantic segmentation by proposing SM-VCENet, a language-model-free framework that enriches visual class embeddings with multi-scale and spatial attention, and it outperforms state-of-the-art methods on benchmarks like PASCAL-5i with relative margins.
Fully supervised semantic segmentation technologies bring a paradigm shift in scene understanding. However, the burden of expensive labeling cost remains as a challenge. To solve the cost problem, recent studies proposed language model based zero-shot semantic segmentation (L-ZSSS) approaches. In this paper, we address L-ZSSS has a limitation in generalization which is a virtue of zero-shot learning. Tackling the limitation, we propose a language-model-free zero-shot semantic segmentation framework, Spatial and Multi-scale aware Visual Class Embedding Network (SM-VCENet). Furthermore, leveraging vision-oriented class embedding SM-VCENet enriches visual information of the class embedding by multi-scale attention and spatial attention. We also propose a novel benchmark (PASCAL2COCO) for zero-shot semantic segmentation, which provides generalization evaluation by domain adaptation and contains visually challenging samples. In experiments, our SM-VCENet outperforms zero-shot semantic segmentation state-of-the-art by a relative margin in PASCAL-5i benchmark and shows generalization-robustness in PASCAL2COCO benchmark.