Robust and Efficient Memory Network for Video Object Segmentation
This work addresses efficiency and robustness issues in semi-supervised video object segmentation, which is important for applications like video editing and autonomous systems, and is incremental by building on existing memory-based methods.
The paper tackles the problem of distractor objects and high computational cost in memory-based video object segmentation by introducing a local attention mechanism and adaptive memory updating, achieving state-of-the-art results with 86.3% J&F on DAVIS 2017 and 85.5% G on YouTube-VOS 2018.
This paper proposes a Robust and Efficient Memory Network, referred to as REMN, for studying semi-supervised video object segmentation (VOS). Memory-based methods have recently achieved outstanding VOS performance by performing non-local pixel-wise matching between the query and memory. However, these methods have two limitations. 1) Non-local matching could cause distractor objects in the background to be incorrectly segmented. 2) Memory features with high temporal redundancy consume significant computing resources. For limitation 1, we introduce a local attention mechanism that tackles the background distraction by enhancing the features of foreground objects with the previous mask. For limitation 2, we first adaptively decide whether to update the memory features depending on the variation of foreground objects to reduce temporal redundancy. Second, we employ a dynamic memory bank, which uses a lightweight and differentiable soft modulation gate to decide how many memory features need to be removed in the temporal dimension. Experiments demonstrate that our REMN achieves state-of-the-art results on DAVIS 2017, with a $\mathcal{J\&F}$ score of 86.3% and on YouTube-VOS 2018, with a $\mathcal{G}$ over mean of 85.5%. Furthermore, our network shows a high inference speed of 25+ FPS and uses relatively few computing resources.