PVO: Panoptic Visual Odometry
This work addresses the need for more comprehensive scene understanding in robotics and autonomous systems by integrating two key tasks, though it appears incremental as it builds on existing methods.
The paper tackles the problem of jointly modeling scene motion, geometry, and panoptic segmentation by proposing PVO, a unified framework for visual odometry and video panoptic segmentation, which outperforms state-of-the-art methods in both tasks.
We present PVO, a novel panoptic visual odometry framework to achieve more comprehensive modeling of the scene motion, geometry, and panoptic segmentation information. Our PVO models visual odometry (VO) and video panoptic segmentation (VPS) in a unified view, which makes the two tasks mutually beneficial. Specifically, we introduce a panoptic update module into the VO Module with the guidance of image panoptic segmentation. This Panoptic-Enhanced VO Module can alleviate the impact of dynamic objects in the camera pose estimation with a panoptic-aware dynamic mask. On the other hand, the VO-Enhanced VPS Module also improves the segmentation accuracy by fusing the panoptic segmentation result of the current frame on the fly to the adjacent frames, using geometric information such as camera pose, depth, and optical flow obtained from the VO Module. These two modules contribute to each other through recurrent iterative optimization. Extensive experiments demonstrate that PVO outperforms state-of-the-art methods in both visual odometry and video panoptic segmentation tasks.