Autoregressive Universal Video Segmentation Model
This addresses the need for efficient, unified video segmentation models for real-world applications like autonomous systems, though it builds incrementally on existing state-space models.
The paper tackles the problem of unprompted video segmentation (detecting and tracking all objects without external cues) by introducing AUSM, a single architecture that unifies both prompted and unprompted segmentation, achieving state-of-the-art performance on multiple benchmarks and up to 2.5x faster training on 16-frame sequences.
Recent video foundation models such as SAM2 excel at prompted video segmentation by treating masks as a general-purpose primitive. However, many real-world settings require unprompted segmentation that aims to detect and track all objects in a video without external cues, leaving today's landscape fragmented across task-specific models and pipelines. We recast streaming video segmentation as sequential mask prediction, analogous to language modeling, and introduce the Autoregressive Universal Segmentation Model (AUSM), a single architecture that unifies both prompted and unprompted video segmentation. Built on recent state-space models, AUSM maintains a fixed-size spatial state and scales to video streams of arbitrary length. Furthermore, all components of AUSM are designed for parallel training across frames, yielding substantial speedups over iterative training. On standard benchmarks (DAVIS17, YouTube-VOS 2018 & 2019, MOSE, YouTube-VIS 2019 & 2021, and OVIS) AUSM outperforms prior universal streaming video segmentation methods and achieves up to 2.5x faster training on 16-frame sequences.