FUTR3D: A Unified Sensor Fusion Framework for 3D Detection
This addresses the need for flexible and efficient sensor fusion in autonomous driving and robotics, offering a novel approach that is not incremental.
The paper tackles the problem of sensor fusion for 3D detection by proposing FUTR3D, a unified framework that works with various sensor configurations, achieving better performance than specifically designed methods and enabling low-cost autonomous driving with competitive results, such as 58.0 mAP using a 4-beam LiDAR and cameras compared to 56.6 mAP with a 32-beam LiDAR.
Sensor fusion is an essential topic in many perception systems, such as autonomous driving and robotics. Existing multi-modal 3D detection models usually involve customized designs depending on the sensor combinations or setups. In this work, we propose the first unified end-to-end sensor fusion framework for 3D detection, named FUTR3D, which can be used in (almost) any sensor configuration. FUTR3D employs a query-based Modality-Agnostic Feature Sampler (MAFS), together with a transformer decoder with a set-to-set loss for 3D detection, thus avoiding using late fusion heuristics and post-processing tricks. We validate the effectiveness of our framework on various combinations of cameras, low-resolution LiDARs, high-resolution LiDARs, and Radars. On NuScenes dataset, FUTR3D achieves better performance over specifically designed methods across different sensor combinations. Moreover, FUTR3D achieves great flexibility with different sensor configurations and enables low-cost autonomous driving. For example, only using a 4-beam LiDAR with cameras, FUTR3D (58.0 mAP) achieves on par performance with state-of-the-art 3D detection model CenterPoint (56.6 mAP) using a 32-beam LiDAR.