Contrastive Language-Colored Pointmap Pretraining for Unified 3D Scene Understanding
This work provides a novel pretraining method for 3D scene understanding that jointly models appearance and geometry, enabling strong performance across multiple downstream tasks with limited data.
UniScene3D introduces a transformer-based encoder that learns unified 3D scene representations from multi-view colored pointmaps by aligning with CLIP, achieving state-of-the-art results in low-shot and task-specific evaluations across viewpoint grounding, scene retrieval, scene type classification, and 3D VQA.
Pretraining 3D encoders by aligning with Contrastive Language Image Pretraining (CLIP) has emerged as a promising direction to learn generalizable representations for 3D scene understanding. In this paper, we propose UniScene3D, a transformer-based encoder that learns unified scene representations from multi-view colored pointmaps, jointly modeling image appearance and geometry. For robust colored pointmap representation learning, we introduce novel cross-view geometric alignment and grounded view alignment to enforce cross-view geometry and semantic consistency. Extensive low-shot and task-specific fine-tuning evaluations on viewpoint grounding, scene retrieval, scene type classification, and 3D VQA demonstrate our state-of-the-art performance. These results highlight the effectiveness of our approach for unified 3D scene understanding. https://yebulabula.github.io/UniScene3D/