G-CUT3R: Guided 3D Reconstruction with Camera and Depth Prior Integration
This addresses 3D reconstruction for computer vision applications, but it is incremental as it builds on an existing model.
The paper tackles the problem of 3D scene reconstruction by enhancing the CUT3R model with prior information like depth and camera data, resulting in significant performance improvements across benchmarks.
We introduce G-CUT3R, a novel feed-forward approach for guided 3D scene reconstruction that enhances the CUT3R model by integrating prior information. Unlike existing feed-forward methods that rely solely on input images, our method leverages auxiliary data, such as depth, camera calibrations, or camera positions, commonly available in real-world scenarios. We propose a lightweight modification to CUT3R, incorporating a dedicated encoder for each modality to extract features, which are fused with RGB image tokens via zero convolution. This flexible design enables seamless integration of any combination of prior information during inference. Evaluated across multiple benchmarks, including 3D reconstruction and other multi-view tasks, our approach demonstrates significant performance improvements, showing its ability to effectively utilize available priors while maintaining compatibility with varying input modalities.