Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-training
This work addresses the challenge of aligning 3D point clouds with text and images for applications like retrieval and captioning, representing an incremental advance in multimodal learning.
The paper tackles the problem of 3D open-world understanding by introducing MixCon3D, a method that sculpts holistic 3D representation through contrastive language-image-3D pre-training, achieving a 5.7% improvement over previous state-of-the-art on the Objaverse-LVIS dataset.
Contrastive learning has emerged as a promising paradigm for 3D open-world understanding, i.e., aligning point cloud representation to image and text embedding space individually. In this paper, we introduce MixCon3D, a simple yet effective method aiming to sculpt holistic 3D representation in contrastive language-image-3D pre-training. In contrast to point cloud only, we develop the 3D object-level representation from complementary perspectives, e.g., multi-view rendered images with the point cloud. Then, MixCon3D performs language-3D contrastive learning, comprehensively depicting real-world 3D objects and bolstering text alignment. Additionally, we pioneer the first thorough investigation of various training recipes for the 3D contrastive learning paradigm, building a solid baseline with improved performance. Extensive experiments conducted on three representative benchmarks reveal that our method significantly improves over the baseline, surpassing the previous state-of-the-art performance on the challenging 1,156-category Objaverse-LVIS dataset by 5.7%. The versatility of MixCon3D is showcased in applications such as text-to-3D retrieval and point cloud captioning, further evidencing its efficacy in diverse scenarios. The code is available at https://github.com/UCSC-VLAA/MixCon3D.