Video-kMaX: A Simple Unified Approach for Online and Near-Online Video Panoptic Segmentation
It addresses the discrepancy between specialized designs for online and near-online video segmentation, offering a unified solution for comprehensive scene understanding in videos.
The paper tackles the problem of unifying online and near-online video panoptic segmentation, proposing Video-kMaX, which achieves state-of-the-art results on datasets like KITTI-STEP, VIPSeg, and VSPW.
Video Panoptic Segmentation (VPS) aims to achieve comprehensive pixel-level scene understanding by segmenting all pixels and associating objects in a video. Current solutions can be categorized into online and near-online approaches. Evolving over the time, each category has its own specialized designs, making it nontrivial to adapt models between different categories. To alleviate the discrepancy, in this work, we propose a unified approach for online and near-online VPS. The meta architecture of the proposed Video-kMaX consists of two components: within clip segmenter (for clip-level segmentation) and cross-clip associater (for association beyond clips). We propose clip-kMaX (clip k-means mask transformer) and HiLA-MB (Hierarchical Location-Aware Memory Buffer) to instantiate the segmenter and associater, respectively. Our general formulation includes the online scenario as a special case by adopting clip length of one. Without bells and whistles, Video-kMaX sets a new state-of-the-art on KITTI-STEP and VIPSeg for video panoptic segmentation, and VSPW for video semantic segmentation. Code will be made publicly available.