Is Contrastive Distillation Enough for Learning Comprehensive 3D Representations?
This addresses a bottleneck in 3D representation learning for computer vision applications, though it appears incremental as it builds on existing contrastive distillation approaches.
The paper tackles the problem of suboptimal 3D representations in cross-modal contrastive distillation by proposing CMCR, a framework that integrates modality-shared and modality-specific features through masked image modeling and occupancy estimation, consistently outperforming existing methods in downstream tasks.
Cross-modal contrastive distillation has recently been explored for learning effective 3D representations. However, existing methods focus primarily on modality-shared features, neglecting the modality-specific features during the pre-training process, which leads to suboptimal representations. In this paper, we theoretically analyze the limitations of current contrastive methods for 3D representation learning and propose a new framework, namely CMCR, to address these shortcomings. Our approach improves upon traditional methods by better integrating both modality-shared and modality-specific features. Specifically, we introduce masked image modeling and occupancy estimation tasks to guide the network in learning more comprehensive modality-specific features. Furthermore, we propose a novel multi-modal unified codebook that learns an embedding space shared across different modalities. Besides, we introduce geometry-enhanced masked image modeling to further boost 3D representation learning. Extensive experiments demonstrate that our method mitigates the challenges faced by traditional approaches and consistently outperforms existing image-to-LiDAR contrastive distillation methods in downstream tasks. Code will be available at https://github.com/Eaphan/CMCR.