Dual Temporal Memory Network for Efficient Video Object Segmentation
This work addresses the problem of improving temporal modeling for researchers and practitioners in video object segmentation, though it appears incremental as it builds on existing methods with hybrid components.
The paper tackles the challenge of leveraging temporal information in Video Object Segmentation (VOS) by proposing a Dual Temporal Memory Network that stores short- and long-term video sequence information, achieving competitive performance on datasets like DAVIS 2016, DAVIS 2017, and Youtube-VOS in terms of speed and accuracy.
Video Object Segmentation (VOS) is typically formulated in a semi-supervised setting. Given the ground-truth segmentation mask on the first frame, the task of VOS is to track and segment the single or multiple objects of interests in the rest frames of the video at the pixel level. One of the fundamental challenges in VOS is how to make the most use of the temporal information to boost the performance. We present an end-to-end network which stores short- and long-term video sequence information preceding the current frame as the temporal memories to address the temporal modeling in VOS. Our network consists of two temporal sub-networks including a short-term memory sub-network and a long-term memory sub-network. The short-term memory sub-network models the fine-grained spatial-temporal interactions between local regions across neighboring frames in video via a graph-based learning framework, which can well preserve the visual consistency of local regions over time. The long-term memory sub-network models the long-range evolution of object via a Simplified-Gated Recurrent Unit (S-GRU), making the segmentation be robust against occlusions and drift errors. In our experiments, we show that our proposed method achieves a favorable and competitive performance on three frequently-used VOS datasets, including DAVIS 2016, DAVIS 2017 and Youtube-VOS in terms of both speed and accuracy.