S4C: Self-Supervised Semantic Scene Completion with Neural Fields
This addresses the challenge of enabling mobile agents to autonomously plan and navigate arbitrary environments by reducing reliance on expensive sensors and manual annotations, though it is incremental as it builds on existing SSC and neural field concepts.
The paper tackles the problem of 3D semantic scene completion (SSC) by proposing S4C, a self-supervised method that reconstructs scenes from single images without relying on costly 3D ground truth data, achieving performance close to fully supervised state-of-the-art methods.
3D semantic scene understanding is a fundamental challenge in computer vision. It enables mobile agents to autonomously plan and navigate arbitrary environments. SSC formalizes this challenge as jointly estimating dense geometry and semantic information from sparse observations of a scene. Current methods for SSC are generally trained on 3D ground truth based on aggregated LiDAR scans. This process relies on special sensors and annotation by hand which are costly and do not scale well. To overcome this issue, our work presents the first self-supervised approach to SSC called S4C that does not rely on 3D ground truth data. Our proposed method can reconstruct a scene from a single image and only relies on videos and pseudo segmentation ground truth generated from off-the-shelf image segmentation network during training. Unlike existing methods, which use discrete voxel grids, we represent scenes as implicit semantic fields. This formulation allows querying any point within the camera frustum for occupancy and semantic class. Our architecture is trained through rendering-based self-supervised losses. Nonetheless, our method achieves performance close to fully supervised state-of-the-art methods. Additionally, our method demonstrates strong generalization capabilities and can synthesize accurate segmentation maps for far away viewpoints.