Re-Nerfing: Improving Novel View Synthesis through Novel View Synthesis
This addresses the challenge of high-quality 3D reconstruction from limited data for applications in computer vision and graphics, but it is an incremental improvement as it builds on existing neural rendering techniques.
The paper tackles the problem of novel view synthesis failing with sparse input images by proposing Re-Nerfing, a method that uses an initial model to generate additional views for training a second model, achieving significant improvements on datasets like mip-NeRF 360 and LLFF.
Recent neural rendering and reconstruction techniques, such as NeRFs or Gaussian Splatting, have shown remarkable novel view synthesis capabilities but require hundreds of images of the scene from diverse viewpoints to render high-quality novel views. With fewer images available, these methods start to fail since they can no longer correctly triangulate the underlying 3D geometry and converge to a non-optimal solution. These failures can manifest as floaters or blurry renderings in sparsely observed areas of the scene. In this paper, we propose Re-Nerfing, a simple and general add-on approach that leverages novel view synthesis itself to tackle this problem. Using an already trained NVS method, we render novel views between existing ones and augment the training data to optimize a second model. This introduces additional multi-view constraints and allows the second model to converge to a better solution. With Re-Nerfing we achieve significant improvements upon multiple pipelines based on NeRF and Gaussian-Splatting in sparse view settings of the mip-NeRF 360 and LLFF datasets. Notably, Re-Nerfing does not require prior knowledge or extra supervision signals, making it a flexible and practical add-on.