PolyphonicFormer: Unified Query Learning for Depth-aware Video Panoptic Segmentation
This addresses the challenge of integrating depth estimation with panoptic segmentation in videos for applications like autonomous driving, offering a novel unified approach to improve robustness and performance.
The paper tackles the Depth-aware Video Panoptic Segmentation (DVPS) problem by proposing PolyphonicFormer, a vision transformer that unifies panoptic segmentation and depth prediction using instance-level depth maps with object queries, achieving state-of-the-art results on Semantic KITTI and Cityscapes datasets and ranking 1st in the ICCV-2021 BMTT Challenge.
The Depth-aware Video Panoptic Segmentation (DVPS) is a new challenging vision problem that aims to predict panoptic segmentation and depth in a video simultaneously. The previous work solves this task by extending the existing panoptic segmentation method with an extra dense depth prediction and instance tracking head. However, the relationship between the depth and panoptic segmentation is not well explored -- simply combining existing methods leads to competition and needs carefully weight balancing. In this paper, we present PolyphonicFormer, a vision transformer to unify these sub-tasks under the DVPS task and lead to more robust results. Our principal insight is that the depth can be harmonized with the panoptic segmentation with our proposed new paradigm of predicting instance level depth maps with object queries. Then the relationship between the two tasks via query-based learning is explored. From the experiments, we demonstrate the benefits of our design from both depth estimation and panoptic segmentation aspects. Since each thing query also encodes the instance-wise information, it is natural to perform tracking directly with appearance learning. Our method achieves state-of-the-art results on two DVPS datasets (Semantic KITTI, Cityscapes), and ranks 1st on the ICCV-2021 BMTT Challenge video + depth track. Code is available at https://github.com/HarborYuan/PolyphonicFormer .