Gaussian Based Adaptive Multi-Modal 3D Semantic Occupancy Prediction
This work addresses safety challenges in autonomous driving by enhancing 3D occupancy prediction, though it appears incremental as it builds on existing multi-modal fusion approaches.
The paper tackles the problem of dense 3D semantic occupancy prediction for autonomous vehicles by proposing a Gaussian-based adaptive multi-modal model that fuses camera and LiDAR data, achieving improved performance with linear computation complexity.
The sparse object detection paradigm shift towards dense 3D semantic occupancy prediction is necessary for dealing with long-tail safety challenges for autonomous vehicles. Nonetheless, the current voxelization methods commonly suffer from excessive computation complexity demands, where the fusion process is brittle, static, and breaks down under dynamic environmental settings. To this end, this research work enhances a novel Gaussian-based adaptive camera-LiDAR multimodal 3D occupancy prediction model that seamlessly bridges the semantic strengths of camera modality with the geometric strengths of LiDAR modality through a memory-efficient 3D Gaussian model. The proposed solution has four key components: (1) LiDAR Depth Feature Aggregation (LDFA), where depth-wise deformable sampling is employed for dealing with geometric sparsity, (2) Entropy-Based Feature Smoothing, where cross-entropy is employed for handling domain-specific noise, (3) Adaptive Camera-LiDAR Fusion, where dynamic recalibration of sensor outputs is performed based on model outputs, and (4) Gauss-Mamba Head that uses Selective State Space Models for global context decoding that enjoys linear computation complexity.