TubeFormer-DeepLab: Video Mask Transformer
This addresses the challenge of divergent models for different video segmentation tasks, offering a unified approach that simplifies models and improves performance.
The paper tackles the problem of unifying multiple video segmentation tasks (semantic, instance, panoptic) by formulating them as assigning labels to video tubes, and presents TubeFormer-DeepLab, which advances state-of-the-art results on multiple benchmarks.
We present TubeFormer-DeepLab, the first attempt to tackle multiple core video segmentation tasks in a unified manner. Different video segmentation tasks (e.g., video semantic/instance/panoptic segmentation) are usually considered as distinct problems. State-of-the-art models adopted in the separate communities have diverged, and radically different approaches dominate in each task. By contrast, we make a crucial observation that video segmentation tasks could be generally formulated as the problem of assigning different predicted labels to video tubes (where a tube is obtained by linking segmentation masks along the time axis) and the labels may encode different values depending on the target task. The observation motivates us to develop TubeFormer-DeepLab, a simple and effective video mask transformer model that is widely applicable to multiple video segmentation tasks. TubeFormer-DeepLab directly predicts video tubes with task-specific labels (either pure semantic categories, or both semantic categories and instance identities), which not only significantly simplifies video segmentation models, but also advances state-of-the-art results on multiple video segmentation benchmarks