SST: Real-time End-to-end Monocular 3D Reconstruction via Sparse Spatial-Temporal Guidance
It addresses the problem of capturing fine structures in real-time 3D reconstruction for applications like robotics and AR/VR, though it appears incremental as it builds on existing end-to-end methods with novel modules.
The paper tackled real-time monocular 3D reconstruction by proposing SST, an end-to-end network that uses sparse spatial guidance and temporal feature fusion to improve detail capture, achieving state-of-the-art results on ScanNet and 7-Scenes datasets with an inference speed of 59 FPS.
Real-time monocular 3D reconstruction is a challenging problem that remains unsolved. Although recent end-to-end methods have demonstrated promising results, tiny structures and geometric boundaries are hardly captured due to their insufficient supervision neglecting spatial details and oversimplified feature fusion ignoring temporal cues. To address the problems, we propose an end-to-end 3D reconstruction network SST, which utilizes Sparse estimated points from visual SLAM system as additional Spatial guidance and fuses Temporal features via a novel cross-modal attention mechanism, achieving more detailed reconstruction results. We propose a Local Spatial-Temporal Fusion module to exploit more informative spatial-temporal cues from multi-view color information and sparse priors, as well a Global Spatial-Temporal Fusion module to refine the local TSDF volumes with the world-frame model from coarse to fine. Extensive experiments on ScanNet and 7-Scenes demonstrate that SST outperforms all state-of-the-art competitors, whilst keeping a high inference speed at 59 FPS, enabling real-world applications with real-time requirements.