PLA: Language-Driven Open-Vocabulary 3D Scene Understanding
It addresses the challenge of localizing and recognizing unseen categories in 3D scenes, which is crucial for robotics and autonomous systems, by overcoming the lack of large-scale 3D-text data.
The paper tackles the problem of open-vocabulary 3D scene understanding by distilling knowledge from pre-trained vision-language models through captioning multi-view images, achieving significant improvements of 25.8% to 44.7% hIoU and 14.5% to 50.4% hAP50 in segmentation tasks.
Open-vocabulary scene understanding aims to localize and recognize unseen categories beyond the annotated label space. The recent breakthrough of 2D open-vocabulary perception is largely driven by Internet-scale paired image-text data with rich vocabulary concepts. However, this success cannot be directly transferred to 3D scenarios due to the inaccessibility of large-scale 3D-text pairs. To this end, we propose to distill knowledge encoded in pre-trained vision-language (VL) foundation models through captioning multi-view images from 3D, which allows explicitly associating 3D and semantic-rich captions. Further, to foster coarse-to-fine visual-semantic representation learning from captions, we design hierarchical 3D-caption pairs, leveraging geometric constraints between 3D scenes and multi-view images. Finally, by employing contrastive learning, the model learns language-aware embeddings that connect 3D and text for open-vocabulary tasks. Our method not only remarkably outperforms baseline methods by 25.8% $\sim$ 44.7% hIoU and 14.5% $\sim$ 50.4% hAP$_{50}$ in open-vocabulary semantic and instance segmentation, but also shows robust transferability on challenging zero-shot domain transfer tasks. See the project website at https://dingry.github.io/projects/PLA.