CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP
This work addresses the labor-intensive annotation bottleneck in 3D scene understanding, enabling open-world recognition for applications like robotics and AR/VR.
The paper tackles the problem of 3D scene understanding without human annotations by transferring CLIP's 2D dense features to 3D models, achieving promising open-vocabulary semantic segmentation results and outperforming previous state-of-the-art methods in zero-shot and data-efficient learning benchmarks.
Training a 3D scene understanding model requires complicated human annotations, which are laborious to collect and result in a model only encoding close-set object semantics. In contrast, vision-language pre-training models (e.g., CLIP) have shown remarkable open-world reasoning properties. To this end, we propose directly transferring CLIP's feature space to 3D scene understanding model without any form of supervision. We first modify CLIP's input and forwarding process so that it can be adapted to extract dense pixel features for 3D scene contents. We then project multi-view image features to the point cloud and train a 3D scene understanding model with feature distillation. Without any annotations or additional training, our model achieves promising annotation-free semantic segmentation results on open-vocabulary semantics and long-tailed concepts. Besides, serving as a cross-modal pre-training framework, our method can be used to improve data efficiency during fine-tuning. Our model outperforms previous SOTA methods in various zero-shot and data-efficient learning benchmarks. Most importantly, our model successfully inherits CLIP's rich-structured knowledge, allowing 3D scene understanding models to recognize not only object concepts but also open-world semantics.