SM3D: Simultaneous Monocular Mapping and 3D Detection
This addresses the challenge of performing mapping and 3D detection together for robotics and autonomous vehicles, offering a novel integration that is not incremental.
The paper tackles the problem of simultaneous mapping and 3D detection in vision-based robotics and self-driving by proposing SM3D, a multi-task deep learning framework that integrates depth estimation and pseudo-LiDAR point clouds, resulting in accuracy improvements of 10.0% and 13.2% over state-of-the-art baselines.
Mapping and 3D detection are two major issues in vision-based robotics, and self-driving. While previous works only focus on each task separately, we present an innovative and efficient multi-task deep learning framework (SM3D) for Simultaneous Mapping and 3D Detection by bridging the gap with robust depth estimation and "Pseudo-LiDAR" point cloud for the first time. The Mapping module takes consecutive monocular frames to generate depth and pose estimation. In 3D Detection module, the depth estimation is projected into 3D space to generate "Pseudo-LiDAR" point cloud, where LiDAR-based 3D detector can be leveraged on point cloud for vehicular 3D detection and localization. By end-to-end training of both modules, the proposed mapping and 3D detection method outperforms the state-of-the-art baseline by 10.0% and 13.2% in accuracy, respectively. While achieving better accuracy, our monocular multi-task SM3D is more than 2 times faster than pure stereo 3D detector, and 18.3% faster than using two modules separately.