Data Augmented 3D Semantic Scene Completion with 2D Segmentation Priors
This work addresses the problem of limited labeled 3D data for training deep networks in SSC, which is important for applications like robotics and assistive computing, but it appears incremental as it builds on existing methods with specific enhancements.
The paper tackles the challenge of semantic scene completion (SSC) by proposing SPAwN, a lightweight multimodal 3D CNN that fuses depth data with 2D segmentation priors and introduces a 3D data augmentation strategy, resulting in consistent performance improvements over previous works with similar complexity.
Semantic scene completion (SSC) is a challenging Computer Vision task with many practical applications, from robotics to assistive computing. Its goal is to infer the 3D geometry in a field of view of a scene and the semantic labels of voxels, including occluded regions. In this work, we present SPAwN, a novel lightweight multimodal 3D deep CNN that seamlessly fuses structural data from the depth component of RGB-D images with semantic priors from a bimodal 2D segmentation network. A crucial difficulty in this field is the lack of fully labeled real-world 3D datasets which are large enough to train the current data-hungry deep 3D CNNs. In 2D computer vision tasks, many data augmentation strategies have been proposed to improve the generalization ability of CNNs. However those approaches cannot be directly applied to the RGB-D input and output volume of SSC solutions. In this paper, we introduce the use of a 3D data augmentation strategy that can be applied to multimodal SSC networks. We validate our contributions with a comprehensive and reproducible ablation study. Our solution consistently surpasses previous works with a similar level of complexity.