DualDiff: Dual-branch Diffusion Model for Autonomous Driving with Semantic Fusion
This work addresses the need for accurate scene reconstruction in autonomous driving, though it appears incremental by enhancing existing diffusion models with semantic fusion techniques.
The paper tackles the problem of generating high-fidelity multi-view driving scenes by addressing limitations in existing methods that use simple 3D bounding boxes and binary maps, resulting in state-of-the-art FID scores and improved performance in downstream tasks like BEV segmentation and 3D object detection.
Accurate and high-fidelity driving scene reconstruction relies on fully leveraging scene information as conditioning. However, existing approaches, which primarily use 3D bounding boxes and binary maps for foreground and background control, fall short in capturing the complexity of the scene and integrating multi-modal information. In this paper, we propose DualDiff, a dual-branch conditional diffusion model designed to enhance multi-view driving scene generation. We introduce Occupancy Ray Sampling (ORS), a semantic-rich 3D representation, alongside numerical driving scene representation, for comprehensive foreground and background control. To improve cross-modal information integration, we propose a Semantic Fusion Attention (SFA) mechanism that aligns and fuses features across modalities. Furthermore, we design a foreground-aware masked (FGM) loss to enhance the generation of tiny objects. DualDiff achieves state-of-the-art performance in FID score, as well as consistently better results in downstream BEV segmentation and 3D object detection tasks.