G3R: Gradient Guided Generalizable Reconstruction
This addresses the need for efficient, high-quality 3D reconstruction for applications like virtual reality and simulation, though it appears incremental as it combines existing optimization and feed-forward methods.
The paper tackles the problem of expensive per-scene optimization and artifacts in large-scale 3D scene reconstruction by introducing G3R, which accelerates reconstruction by at least 10x while achieving comparable or better realism than 3DGS and improving robustness to view changes.
Large scale 3D scene reconstruction is important for applications such as virtual reality and simulation. Existing neural rendering approaches (e.g., NeRF, 3DGS) have achieved realistic reconstructions on large scenes, but optimize per scene, which is expensive and slow, and exhibit noticeable artifacts under large view changes due to overfitting. Generalizable approaches or large reconstruction models are fast, but primarily work for small scenes/objects and often produce lower quality rendering results. In this work, we introduce G3R, a generalizable reconstruction approach that can efficiently predict high-quality 3D scene representations for large scenes. We propose to learn a reconstruction network that takes the gradient feedback signals from differentiable rendering to iteratively update a 3D scene representation, combining the benefits of high photorealism from per-scene optimization with data-driven priors from fast feed-forward prediction methods. Experiments on urban-driving and drone datasets show that G3R generalizes across diverse large scenes and accelerates the reconstruction process by at least 10x while achieving comparable or better realism compared to 3DGS, and also being more robust to large view changes.