Unified Embedding Alignment for Open-Vocabulary Video Instance Segmentation
This addresses the domain gap and temporal underutilization in open-vocabulary video segmentation, offering incremental improvements for computer vision applications.
The paper tackles the problem of Open-Vocabulary Video Instance Segmentation (VIS), where existing methods have poor generalization to novel categories, by proposing OVFormer which aligns query embeddings with CLIP features and leverages temporal consistency through video-based training. It achieves state-of-the-art results, including 21.9 mAP on LV-VIS (7.7 mAP improvement) and strong zero-shot gains on other datasets.
Open-Vocabulary Video Instance Segmentation (VIS) is attracting increasing attention due to its ability to segment and track arbitrary objects. However, the recent Open-Vocabulary VIS attempts obtained unsatisfactory results, especially in terms of generalization ability of novel categories. We discover that the domain gap between the VLM features (e.g., CLIP) and the instance queries and the underutilization of temporal consistency are two central causes. To mitigate these issues, we design and train a novel Open-Vocabulary VIS baseline called OVFormer. OVFormer utilizes a lightweight module for unified embedding alignment between query embeddings and CLIP image embeddings to remedy the domain gap. Unlike previous image-based training methods, we conduct video-based model training and deploy a semi-online inference scheme to fully mine the temporal consistency in the video. Without bells and whistles, OVFormer achieves 21.9 mAP with a ResNet-50 backbone on LV-VIS, exceeding the previous state-of-the-art performance by 7.7. Extensive experiments on some Close-Vocabulary VIS datasets also demonstrate the strong zero-shot generalization ability of OVFormer (+ 7.6 mAP on YouTube-VIS 2019, + 3.9 mAP on OVIS). Code is available at https://github.com/fanghaook/OVFormer.