VISAGE: Video Instance Segmentation with Appearance-Guided Enhancement
This work addresses a key limitation in video instance segmentation for computer vision applications, offering an incremental improvement to existing methods.
The paper tackles the problem of incorrect object associations in video instance segmentation due to over-reliance on location information, and by enhancing object decoders to capture appearance embeddings, it achieves state-of-the-art results on YouTube-VIS 2019/2021 and Occluded VIS benchmarks.
In recent years, online Video Instance Segmentation (VIS) methods have shown remarkable advancement with their powerful query-based detectors. Utilizing the output queries of the detector at the frame-level, these methods achieve high accuracy on challenging benchmarks. However, our observations demonstrate that these methods heavily rely on location information, which often causes incorrect associations between objects. This paper presents that a key axis of object matching in trackers is appearance information, which becomes greatly instructive under conditions where positional cues are insufficient for distinguishing their identities. Therefore, we suggest a simple yet powerful extension to object decoders that explicitly extract embeddings from backbone features and drive queries to capture the appearances of objects, which greatly enhances instance association accuracy. Furthermore, recognizing the limitations of existing benchmarks in fully evaluating appearance awareness, we have constructed a synthetic dataset to rigorously validate our method. By effectively resolving the over-reliance on location information, we achieve state-of-the-art results on YouTube-VIS 2019/2021 and Occluded VIS (OVIS). Code is available at https://github.com/KimHanjung/VISAGE.