CAT-Det: Contrastively Augmented Transformer for Multi-modal 3D Object Detection
This addresses the problem of multi-modal 3D object detection for autonomous driving systems, representing an incremental improvement with novel components.
The paper tackled the challenge of insufficiently using LiDAR and RGB data due to inter-modal discrepancies in 3D object detection for autonomous driving, proposing CAT-Det with a contrastive augmentation approach that achieved a new state-of-the-art on the KITTI benchmark.
In autonomous driving, LiDAR point-clouds and RGB images are two major data modalities with complementary cues for 3D object detection. However, it is quite difficult to sufficiently use them, due to large inter-modal discrepancies. To address this issue, we propose a novel framework, namely Contrastively Augmented Transformer for multi-modal 3D object Detection (CAT-Det). Specifically, CAT-Det adopts a two-stream structure consisting of a Pointformer (PT) branch, an Imageformer (IT) branch along with a Cross-Modal Transformer (CMT) module. PT, IT and CMT jointly encode intra-modal and inter-modal long-range contexts for representing an object, thus fully exploring multi-modal information for detection. Furthermore, we propose an effective One-way Multi-modal Data Augmentation (OMDA) approach via hierarchical contrastive learning at both the point and object levels, significantly improving the accuracy only by augmenting point-clouds, which is free from complex generation of paired samples of the two modalities. Extensive experiments on the KITTI benchmark show that CAT-Det achieves a new state-of-the-art, highlighting its effectiveness.