PMVOS: Pixel-Level Matching-Based Video Object Segmentation
This work addresses the limitation of using only initial and previous frames for template features in video object segmentation, offering an incremental improvement for researchers and practitioners in computer vision.
The paper tackles the problem of semi-supervised video object segmentation by proposing PMVOS, which constructs template features using information from all past frames and applies self-attention to similarity maps, achieving state-of-the-art real-time performance with a J&F score of 85.6% on DAVIS 2016.
Semi-supervised video object segmentation (VOS) aims to segment arbitrary target objects in video when the ground truth segmentation mask of the initial frame is provided. Due to this limitation of using prior knowledge about the target object, feature matching, which compares template features representing the target object with input features, is an essential step. Recently, pixel-level matching (PM), which matches every pixel in template features and input features, has been widely used for feature matching because of its high performance. However, despite its effectiveness, the information used to build the template features is limited to the initial and previous frames. We address this issue by proposing a novel method-PM-based video object segmentation (PMVOS)-that constructs strong template features containing the information of all past frames. Furthermore, we apply self-attention to the similarity maps generated from PM to capture global dependencies. On the DAVIS 2016 validation set, we achieve new state-of-the-art performance among real-time methods (> 30 fps), with a J&F score of 85.6%. Performance on the DAVIS 2017 and YouTube-VOS validation sets is also impressive, with J&F scores of 74.0% and 68.2%, respectively.