Learning Neural Radiance Fields from Multi-View Geometry
This incremental improvement addresses geometry quality issues in NeRF for 3D reconstruction, benefiting researchers and practitioners in computer vision and graphics.
The paper tackles the problem of noisy and incorrect geometry in Neural Radiance Fields (NeRF) when extracting meshes, by integrating classical Multi-View Geometry algorithms to provide geometric priors, resulting in cleaner 3D meshes while maintaining competitive novel view synthesis performance.
We present a framework, called MVG-NeRF, that combines classical Multi-View Geometry algorithms and Neural Radiance Fields (NeRF) for image-based 3D reconstruction. NeRF has revolutionized the field of implicit 3D representations, mainly due to a differentiable volumetric rendering formulation that enables high-quality and geometry-aware novel view synthesis. However, the underlying geometry of the scene is not explicitly constrained during training, thus leading to noisy and incorrect results when extracting a mesh with marching cubes. To this end, we propose to leverage pixelwise depths and normals from a classical 3D reconstruction pipeline as geometric priors to guide NeRF optimization. Such priors are used as pseudo-ground truth during training in order to improve the quality of the estimated underlying surface. Moreover, each pixel is weighted by a confidence value based on the forward-backward reprojection error for additional robustness. Experimental results on real-world data demonstrate the effectiveness of this approach in obtaining clean 3D meshes from images, while maintaining competitive performances in novel view synthesis.