NeuSD: Surface Completion with Multi-View Text-to-Image Diffusion
This addresses the challenge of incomplete 3D surface reconstruction for computer vision and graphics applications, representing an incremental advance by combining existing techniques with new components.
The paper tackles the problem of 3D surface reconstruction from multiple images where only part of the object is captured, by using neural radiance fields for visible parts and a novel method to complete unobserved regions with multi-view text-to-image diffusion, resulting in significant qualitative and quantitative improvements on the BlendedMVS dataset.
We present a novel method for 3D surface reconstruction from multiple images where only a part of the object of interest is captured. Our approach builds on two recent developments: surface reconstruction using neural radiance fields for the reconstruction of the visible parts of the surface, and guidance of pre-trained 2D diffusion models in the form of Score Distillation Sampling (SDS) to complete the shape in unobserved regions in a plausible manner. We introduce three components. First, we suggest employing normal maps as a pure geometric representation for SDS instead of color renderings which are entangled with the appearance information. Second, we introduce the freezing of the SDS noise during training which results in more coherent gradients and better convergence. Third, we propose Multi-View SDS as a way to condition the generation of the non-observable part of the surface without fine-tuning or making changes to the underlying 2D Stable Diffusion model. We evaluate our approach on the BlendedMVS dataset demonstrating significant qualitative and quantitative improvements over competing methods.