Open-vocabulary Panoptic Segmentation with Embedding Modulation
This addresses the problem of segmenting novel objects in images for real-world applications, offering a data-efficient solution that improves over previous methods.
The paper tackles open-vocabulary panoptic segmentation by proposing OPSNet, which uses an Embedding Modulation module to enhance embeddings and exchange information with CLIP, achieving state-of-the-art results on multiple datasets like COCO and ADE20K with reduced need for extra data.
Open-vocabulary image segmentation is attracting increasing attention due to its critical applications in the real world. Traditional closed-vocabulary segmentation methods are not able to characterize novel objects, whereas several recent open-vocabulary attempts obtain unsatisfactory results, i.e., notable performance reduction on the closed vocabulary and massive demand for extra data. To this end, we propose OPSNet, an omnipotent and data-efficient framework for Open-vocabulary Panoptic Segmentation. Specifically, the exquisitely designed Embedding Modulation module, together with several meticulous components, enables adequate embedding enhancement and information exchange between the segmentation model and the visual-linguistic well-aligned CLIP encoder, resulting in superior segmentation performance under both open- and closed-vocabulary settings with much fewer need of additional data. Extensive experimental evaluations are conducted across multiple datasets (e.g., COCO, ADE20K, Cityscapes, and PascalContext) under various circumstances, where the proposed OPSNet achieves state-of-the-art results, which demonstrates the effectiveness and generality of the proposed approach. The code and trained models will be made publicly available.