Robust Single-shot Structured Light 3D Imaging via Neural Feature Decoding
This work addresses robustness issues in 3D sensing for applications like consumer devices (e.g., Apple Face ID), though it is incremental as it builds on neural feature matching and monocular depth estimation.
The paper tackles the problem of robust 3D imaging in single-shot structured light systems, which often fail under occlusions or non-Lambertian surfaces, by proposing a learning-based framework that decodes depth correspondences in neural feature space and refines depth using monocular priors, achieving substantial performance improvements over traditional methods and outperforming commercial systems and passive stereo methods in real-world indoor environments.
We consider the problem of active 3D imaging using single-shot structured light systems, which are widely employed in commercial 3D sensing devices such as Apple Face ID and Intel RealSense. Traditional structured light methods typically decode depth correspondences through pixel-domain matching algorithms, resulting in limited robustness under challenging scenarios like occlusions, fine-structured details, and non-Lambertian surfaces. Inspired by recent advances in neural feature matching, we propose a learning-based structured light decoding framework that performs robust correspondence matching within feature space rather than the fragile pixel domain. Our method extracts neural features from the projected patterns and captured infrared (IR) images, explicitly incorporating their geometric priors by building cost volumes in feature space, achieving substantial performance improvements over pixel-domain decoding approaches. To further enhance depth quality, we introduce a depth refinement module that leverages strong priors from large-scale monocular depth estimation models, improving fine detail recovery and global structural coherence. To facilitate effective learning, we develop a physically-based structured light rendering pipeline, generating nearly one million synthetic pattern-image pairs with diverse objects and materials for indoor settings. Experiments demonstrate that our method, trained exclusively on synthetic data with multiple structured light patterns, generalizes well to real-world indoor environments, effectively processes various pattern types without retraining, and consistently outperforms both commercial structured light systems and passive stereo RGB-based depth estimation methods. Project page: https://namisntimpot.github.io/NSLweb/.