FSMDet: Vision-guided feature diffusion for fully sparse 3D detector
This addresses efficiency and performance challenges in 3D object detection for autonomous driving and robotics, representing a novel hybrid approach rather than a foundational breakthrough.
The paper tackles the problem of limited feature diffusion in fully sparse 3D detection by proposing FSMDet, which uses visual information to guide LiDAR feature diffusion while maintaining efficiency. The method improves performance over LiDAR-only fully sparse models, achieves state-of-the-art results in multimodal models, and is up to 5 times more efficient in inference than previous SOTA methods.
Fully sparse 3D detection has attracted an increasing interest in the recent years. However, the sparsity of the features in these frameworks challenges the generation of proposals because of the limited diffusion process. In addition, the quest for efficiency has led to only few work on vision-assisted fully sparse models. In this paper, we propose FSMDet (Fully Sparse Multi-modal Detection), which use visual information to guide the LiDAR feature diffusion process while still maintaining the efficiency of the pipeline. Specifically, most of fully sparse works focus on complex customized center fusion diffusion/regression operators. However, we observed that if the adequate object completion is performed, even the simplest interpolation operator leads to satisfactory results. Inspired by this observation, we split the vision-guided diffusion process into two modules: a Shape Recover Layer (SRLayer) and a Self Diffusion Layer (SDLayer). The former uses RGB information to recover the shape of the visible part of an object, and the latter uses a visual prior to further spread the features to the center region. Experiments demonstrate that our approach successfully improves the performance of previous fully sparse models that use LiDAR only and reaches SOTA performance in multimodal models. At the same time, thanks to the sparse architecture, our method can be up to 5 times more efficient than previous SOTA methods in the inference process.