MonoPatchNeRF: Improving Neural Radiance Fields with Patch-based Monocular Guidance
This work addresses geometry accuracy issues in NeRF for sparse view 3D reconstruction, which is incremental as it builds on existing density-based methods with patch-based constraints.
The paper tackles the problem of poor geometry and view extrapolation in regularized Neural Radiance Fields (NeRF) for large-scale sparse view scenes like ETH3D by using a density-based approach with patch-based monocular guidance. The result is a significant improvement in geometric accuracy, such as increasing the F1@2cm score by 4x-8x compared to other methods, while maintaining similar view synthesis quality and reducing training and inference time.
The latest regularized Neural Radiance Field (NeRF) approaches produce poor geometry and view extrapolation for large scale sparse view scenes, such as ETH3D. Density-based approaches tend to be under-constrained, while surface-based approaches tend to miss details. In this paper, we take a density-based approach, sampling patches instead of individual rays to better incorporate monocular depth and normal estimates and patch-based photometric consistency constraints between training views and sampled virtual views. Loosely constraining densities based on estimated depth aligned to sparse points further improves geometric accuracy. While maintaining similar view synthesis quality, our approach significantly improves geometric accuracy on the ETH3D benchmark, e.g. increasing the F1@2cm score by 4x-8x compared to other regularized density-based approaches, with much lower training and inference time than other approaches.