Hierarchical Spatiotemporal Transformers for Video Object Segmentation
This work addresses the problem of accurately segmenting objects in videos, which is crucial for applications like video editing and autonomous systems, but it is incremental as it builds on existing transformer architectures.
The paper tackles semi-supervised video object segmentation by proposing HST, a framework that uses hierarchical spatiotemporal transformers to extract and combine image and video features, achieving state-of-the-art results on benchmarks like YouTube-VOS (85.0%), DAVIS 2017 (85.9%), and DAVIS 2016 (94.0%).
This paper presents a novel framework called HST for semi-supervised video object segmentation (VOS). HST extracts image and video features using the latest Swin Transformer and Video Swin Transformer to inherit their inductive bias for the spatiotemporal locality, which is essential for temporally coherent VOS. To take full advantage of the image and video features, HST casts image and video features as a query and memory, respectively. By applying efficient memory read operations at multiple scales, HST produces hierarchical features for the precise reconstruction of object masks. HST shows effectiveness and robustness in handling challenging scenarios with occluded and fast-moving objects under cluttered backgrounds. In particular, HST-B outperforms the state-of-the-art competitors on multiple popular benchmarks, i.e., YouTube-VOS (85.0%), DAVIS 2017 (85.9%), and DAVIS 2016 (94.0%).