Technical Report: Co-learning of geometry and semantics for online 3D mapping
This work addresses semantic 3D mapping for autonomous navigation, but it appears incremental as it builds on existing multi-task learning and stereo vision techniques.
The paper tackles 3D semantic reconstruction for autonomous navigation by co-learning depth and semantic segmentation, resulting in a method that repeatedly surpasses state-of-the-art approaches on the 3DRMS Challenge dataset.
This paper is a technical report about our submission for the ECCV 2018 3DRMS Workshop Challenge on Semantic 3D Reconstruction \cite{Tylecek2018rms}. In this paper, we address 3D semantic reconstruction for autonomous navigation using co-learning of depth map and semantic segmentation. The core of our pipeline is a deep multi-task neural network which tightly refines depth and also produces accurate semantic segmentation maps. Its inputs are an image and a raw depth map produced from a pair of images by standard stereo vision. The resulting semantic 3D point clouds are then merged in order to create a consistent 3D mesh, in turn used to produce dense semantic 3D reconstruction maps. The performances of each step of the proposed method are evaluated on the dataset and multiple tasks of the 3DRMS Challenge, and repeatedly surpass state-of-the-art approaches.