RI3D: Few-Shot Gaussian Splatting With Repair and Inpainting Diffusion Priors
This addresses the challenge of few-shot 3D reconstruction for applications like virtual reality and robotics, though it appears incremental as it builds on existing 3DGS and diffusion model techniques.
The paper tackles the problem of reconstructing high-quality 3D novel views from extremely sparse input images by proposing RI3D, which separates view synthesis into reconstructing visible regions and hallucinating missing regions using two personalized diffusion models. The approach outperforms state-of-the-art methods on diverse scenes with sparse inputs.
In this paper, we propose RI3D, a novel 3DGS-based approach that harnesses the power of diffusion models to reconstruct high-quality novel views given a sparse set of input images. Our key contribution is separating the view synthesis process into two tasks of reconstructing visible regions and hallucinating missing regions, and introducing two personalized diffusion models, each tailored to one of these tasks. Specifically, one model ('repair') takes a rendered image as input and predicts the corresponding high-quality image, which in turn is used as a pseudo ground truth image to constrain the optimization. The other model ('inpainting') primarily focuses on hallucinating details in unobserved areas. To integrate these models effectively, we introduce a two-stage optimization strategy: the first stage reconstructs visible areas using the repair model, and the second stage reconstructs missing regions with the inpainting model while ensuring coherence through further optimization. Moreover, we augment the optimization with a novel Gaussian initialization method that obtains per-image depth by combining 3D-consistent and smooth depth with highly detailed relative depth. We demonstrate that by separating the process into two tasks and addressing them with the repair and inpainting models, we produce results with detailed textures in both visible and missing regions that outperform state-of-the-art approaches on a diverse set of scenes with extremely sparse inputs.