Zero-Shot 4D Lidar Panoptic Segmentation
This addresses the challenge of segmenting and recognizing arbitrary objects in Lidar for embodied navigation, with applications in streaming perception and semantic mapping, though it is incremental as it builds on existing models.
The paper tackles the problem of zero-shot 4D Lidar panoptic segmentation by proposing SAL-4D, which uses multi-modal sensor setups to distill video object segmentation and vision-language models into Lidar, achieving over 5 PQ improvement in 3D zero-shot Lidar panoptic segmentation and enabling zero-shot 4D-LPS.
Zero-shot 4D segmentation and recognition of arbitrary objects in Lidar is crucial for embodied navigation, with applications ranging from streaming perception to semantic mapping and localization. However, the primary challenge in advancing research and developing generalized, versatile methods for spatio-temporal scene understanding in Lidar lies in the scarcity of datasets that provide the necessary diversity and scale of annotations.To overcome these challenges, we propose SAL-4D (Segment Anything in Lidar--4D), a method that utilizes multi-modal robotic sensor setups as a bridge to distill recent developments in Video Object Segmentation (VOS) in conjunction with off-the-shelf Vision-Language foundation models to Lidar. We utilize VOS models to pseudo-label tracklets in short video sequences, annotate these tracklets with sequence-level CLIP tokens, and lift them to the 4D Lidar space using calibrated multi-modal sensory setups to distill them to our SAL-4D model. Due to temporal consistent predictions, we outperform prior art in 3D Zero-Shot Lidar Panoptic Segmentation (LPS) over $5$ PQ, and unlock Zero-Shot 4D-LPS.