Toward Unified Multimodal Representation Learning for Autonomous Driving
This work addresses the problem of inconsistent multimodal alignment for autonomous driving systems by proposing a unified approach, which could lead to more robust end-to-end autonomous driving.
The paper proposes a Contrastive Tensor Pre-training (CTP) framework to align multiple modalities (text, image, point cloud) in a unified embedding space for autonomous driving. This method extends pairwise cosine similarity to a multimodal similarity tensor and introduces a tensor loss for joint contrastive learning. The framework achieves favorable performance when aligning a 3D encoder with pretrained CLIP encoders and when pretraining all encoders from scratch.
Contrastive Language-Image Pre-training (CLIP) has shown impressive performance in aligning visual and textual representations. Recent studies have extended this paradigm to 3D vision to improve scene understanding for autonomous driving. A common strategy is to employ pairwise cosine similarity between modalities to guide the training of a 3D encoder. However, considering the similarity between individual modality pairs rather than all modalities jointly fails to ensure consistent and unified alignment across the entire multimodal space. In this paper, we propose a Contrastive Tensor Pre-training (CTP) framework that simultaneously aligns multiple modalities in a unified embedding space to enhance end-to-end autonomous driving. Compared with pairwise cosine similarity alignment, our method extends the 2D similarity matrix into a multimodal similarity tensor. Furthermore, we introduce a tensor loss to enable joint contrastive learning across all modalities. For experimental validation of our framework, we construct a text-image-point cloud triplet dataset derived from existing autonomous driving datasets. The results show that our proposed unified multimodal alignment framework achieves favorable performance for both scenarios: (i) aligning a 3D encoder with pretrained CLIP encoders, and (ii) pretraining all encoders from scratch.