MonoDVPS: A Self-Supervised Monocular Depth Estimation Approach to Depth-aware Video Panoptic Segmentation
This work addresses the problem of reconstructing 3D panoptic point clouds from video for applications like autonomous driving, but it is incremental as it builds on existing methods with specific improvements.
The paper tackles depth-aware video panoptic segmentation by proposing a multi-task network that combines self-supervised monocular depth estimation with semi-supervised video panoptic segmentation, achieving competitive results on Cityscapes-DVPS and SemKITTI-DVPS datasets with fast inference speed.
Depth-aware video panoptic segmentation tackles the inverse projection problem of restoring panoptic 3D point clouds from video sequences, where the 3D points are augmented with semantic classes and temporally consistent instance identifiers. We propose a novel solution with a multi-task network that performs monocular depth estimation and video panoptic segmentation. Since acquiring ground truth labels for both depth and image segmentation has a relatively large cost, we leverage the power of unlabeled video sequences with self-supervised monocular depth estimation and semi-supervised learning from pseudo-labels for video panoptic segmentation. To further improve the depth prediction, we introduce panoptic-guided depth losses and a novel panoptic masking scheme for moving objects to avoid corrupting the training signal. Extensive experiments on the Cityscapes-DVPS and SemKITTI-DVPS datasets demonstrate that our model with the proposed improvements achieves competitive results and fast inference speed.