Neural Rendering based Urban Scene Reconstruction for Autonomous Driving
This addresses the problem of enhancing 3D scene reconstruction for autonomous driving applications, such as annotation and data augmentation, but is incremental as it builds on existing neural rendering methods.
The paper tackles dense 3D reconstruction for autonomous driving by combining LiDAR and camera data using neural implicit surfaces and radiance fields, resulting in accurate and dense 3D structures with qualitative and quantitative improvements on automotive scenes.
Dense 3D reconstruction has many applications in automated driving including automated annotation validation, multimodal data augmentation, providing ground truth annotations for systems lacking LiDAR, as well as enhancing auto-labeling accuracy. LiDAR provides highly accurate but sparse depth, whereas camera images enable estimation of dense depth but noisy particularly at long ranges. In this paper, we harness the strengths of both sensors and propose a multimodal 3D scene reconstruction using a framework combining neural implicit surfaces and radiance fields. In particular, our method estimates dense and accurate 3D structures and creates an implicit map representation based on signed distance fields, which can be further rendered into RGB images, and depth maps. A mesh can be extracted from the learned signed distance field and culled based on occlusion. Dynamic objects are efficiently filtered on the fly during sampling using 3D object detection models. We demonstrate qualitative and quantitative results on challenging automotive scenes.