3D Reconstruction with Generalizable Neural Fields using Scene Priors
This addresses scalability and flexibility issues in 3D reconstruction for applications requiring fast adaptation with fewer views, though it builds incrementally on existing neural field methods.
The paper tackles the problem of inefficient and view-limited 3D scene reconstruction by introducing generalizable Neural Fields with scene priors (NFPs), which map single-view RGB-D images into signed distance and radiance values, achieving state-of-the-art performance and efficiency while enabling single-image novel-view synthesis.
High-fidelity 3D scene reconstruction has been substantially advanced by recent progress in neural fields. However, most existing methods train a separate network from scratch for each individual scene. This is not scalable, inefficient, and unable to yield good results given limited views. While learning-based multi-view stereo methods alleviate this issue to some extent, their multi-view setting makes it less flexible to scale up and to broad applications. Instead, we introduce training generalizable Neural Fields incorporating scene Priors (NFPs). The NFP network maps any single-view RGB-D image into signed distance and radiance values. A complete scene can be reconstructed by merging individual frames in the volumetric space WITHOUT a fusion module, which provides better flexibility. The scene priors can be trained on large-scale datasets, allowing for fast adaptation to the reconstruction of a new scene with fewer views. NFP not only demonstrates SOTA scene reconstruction performance and efficiency, but it also supports single-image novel-view synthesis, which is underexplored in neural fields. More qualitative results are available at: https://oasisyang.github.io/neural-prior