TALO: Pushing 3D Vision Foundation Models Towards Globally Consistent Online Reconstruction
This addresses a key limitation for applications like autonomous driving by improving consistency in online 3D reconstruction, though it is incremental as it builds on existing foundation models.
The paper tackles the problem of maintaining temporal consistency in 3D vision foundation models for online reconstruction, proposing a framework that uses Thin Plate Spline and submap registration to achieve more coherent geometry and lower trajectory errors across multiple datasets and setups.
3D vision foundation models have shown strong generalization in reconstructing key 3D attributes from uncalibrated images through a single feed-forward pass. However, when deployed in online settings such as driving scenarios, predictions are made over temporal windows, making it non-trivial to maintain consistency across time. Recent strategies align consecutive predictions by solving global transformation, yet our analysis reveals their fundamental limitations in assumption validity, local alignment scope, and robustness under noisy geometry. In this work, we propose a higher-DOF and long-term alignment framework based on Thin Plate Spline, leveraging globally propagated control points to correct spatially varying inconsistencies. In addition, we adopt a point-agnostic submap registration design that is inherently robust to noisy geometry predictions. The proposed framework is fully plug-and-play, compatible with diverse 3D foundation models and camera configurations (e.g., monocular or surround-view). Extensive experiments demonstrate that our method consistently yields more coherent geometry and lower trajectory errors across multiple datasets, backbone models, and camera setups, highlighting its robustness and generality. Codes are publicly available at \href{https://github.com/Xian-Bei/TALO}{https://github.com/Xian-Bei/TALO}.