See More and Know More: Zero-shot Point Cloud Segmentation via Multi-modal Visual Data
It addresses the problem of recognizing novel objects in point clouds for applications like autonomous driving, but it is incremental as it builds on existing zero-shot learning pipelines by adding multi-modal data.
The paper tackles zero-shot point cloud segmentation by proposing a multi-modal method that combines point clouds with images to improve visual-semantic alignment, achieving 52% and 49% average mIoU improvements on unseen classes in SemanticKITTI and nuScenes benchmarks.
Zero-shot point cloud segmentation aims to make deep models capable of recognizing novel objects in point cloud that are unseen in the training phase. Recent trends favor the pipeline which transfers knowledge from seen classes with labels to unseen classes without labels. They typically align visual features with semantic features obtained from word embedding by the supervision of seen classes' annotations. However, point cloud contains limited information to fully match with semantic features. In fact, the rich appearance information of images is a natural complement to the textureless point cloud, which is not well explored in previous literature. Motivated by this, we propose a novel multi-modal zero-shot learning method to better utilize the complementary information of point clouds and images for more accurate visual-semantic alignment. Extensive experiments are performed in two popular benchmarks, i.e., SemanticKITTI and nuScenes, and our method outperforms current SOTA methods with 52% and 49% improvement on average for unseen class mIoU, respectively.