XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model
This addresses the challenge of memory consumption and accuracy decay in long-term video object segmentation, offering a novel solution for applications requiring processing of extended video sequences.
The paper tackles the problem of video object segmentation in long videos by introducing XMem, an architecture inspired by the Atkinson-Shiffrin memory model with multiple feature memory stores, which greatly exceeds state-of-the-art performance on long-video datasets while matching methods on short-video datasets.
We present XMem, a video object segmentation architecture for long videos with unified feature memory stores inspired by the Atkinson-Shiffrin memory model. Prior work on video object segmentation typically only uses one type of feature memory. For videos longer than a minute, a single feature memory model tightly links memory consumption and accuracy. In contrast, following the Atkinson-Shiffrin model, we develop an architecture that incorporates multiple independent yet deeply-connected feature memory stores: a rapidly updated sensory memory, a high-resolution working memory, and a compact thus sustained long-term memory. Crucially, we develop a memory potentiation algorithm that routinely consolidates actively used working memory elements into the long-term memory, which avoids memory explosion and minimizes performance decay for long-term prediction. Combined with a new memory reading mechanism, XMem greatly exceeds state-of-the-art performance on long-video datasets while being on par with state-of-the-art methods (that do not work on long videos) on short-video datasets. Code is available at https://hkchengrex.github.io/XMem