RGB-X Object Detection via Scene-Specific Fusion Modules
This work addresses the challenge of enabling robust visual understanding in all weather conditions for autonomous vehicles, representing an incremental improvement by reducing data and parameter requirements compared to existing methods.
The paper tackles the problem of multimodal sensor fusion for autonomous vehicles by proposing an efficient RGB-X fusion network that uses scene-specific modules to combine pretrained single-modal models, achieving superior performance on RGB-thermal and RGB-gated datasets with minimal additional parameters.
Multimodal deep sensor fusion has the potential to enable autonomous vehicles to visually understand their surrounding environments in all weather conditions. However, existing deep sensor fusion methods usually employ convoluted architectures with intermingled multimodal features, requiring large coregistered multimodal datasets for training. In this work, we present an efficient and modular RGB-X fusion network that can leverage and fuse pretrained single-modal models via scene-specific fusion modules, thereby enabling joint input-adaptive network architectures to be created using small, coregistered multimodal datasets. Our experiments demonstrate the superiority of our method compared to existing works on RGB-thermal and RGB-gated datasets, performing fusion using only a small amount of additional parameters. Our code is available at https://github.com/dsriaditya999/RGBXFusion.