Mono-STAR: Mono-camera Scene-level Tracking and Reconstruction
This addresses the need for robust real-time 3D scene understanding in applications like robotics and AR/VR, though it appears incremental as it builds on existing reconstruction frameworks.
The authors tackled the problem of real-time 3D reconstruction from monocular cameras by developing Mono-STAR, which simultaneously handles semantic fusion, fast motion tracking, non-rigid deformation, and topological changes. The system significantly outperformed existing state-of-the-art methods in challenging scenes.
We present Mono-STAR, the first real-time 3D reconstruction system that simultaneously supports semantic fusion, fast motion tracking, non-rigid object deformation, and topological change under a unified framework. The proposed system solves a new optimization problem incorporating optical-flow-based 2D constraints to deal with fast motion and a novel semantic-aware deformation graph (SAD-graph) for handling topology change. We test the proposed system under various challenging scenes and demonstrate that it significantly outperforms existing state-of-the-art methods.