S-MUSt3R: Sliding Multi-view 3D Reconstruction
This work addresses scalability issues in 3D reconstruction for applications like robot navigation, though it appears incremental as it builds on the existing MUSt3R model with a new processing strategy.
The paper tackles the challenge of extending foundation models to large-scale monocular 3D reconstruction from RGB streams, which is limited by memory constraints, and proposes S-MUSt3R, a pipeline that achieves trajectory and reconstruction performance comparable to traditional methods without model retraining.
The recent paradigm shift in 3D vision led to the rise of foundation models with remarkable capabilities in 3D perception from uncalibrated images. However, extending these models to large-scale RGB stream 3D reconstruction remains challenging due to memory limitations. This work proposes S-MUSt3R, a simple and efficient pipeline that extends the limits of foundation models for monocular 3D reconstruction. Our approach addresses the scalability bottleneck of foundation models through a simple strategy of sequence segmentation followed by segment alignment and lightweight loop closure optimization. Without model retraining, we benefit from remarkable 3D reconstruction capacities of MUSt3R model and achieve trajectory and reconstruction performance comparable to traditional methods with more complex architecture. We evaluate S-MUSt3R on TUM, 7-Scenes and proprietary robot navigation datasets and show that S-MUSt3R runs successfully on long RGB sequences and produces accurate and consistent 3D reconstruction. Our results highlight the potential of leveraging the MUSt3R model for scalable monocular 3D scene in real-world settings, with an important advantage of making predictions directly in the metric space.