SplitNeRF: Split Sum Approximation Neural Field for Joint Geometry, Illumination, and Material Estimation
This work addresses the challenge of efficient and accurate 3D reconstruction and relighting for applications in computer graphics and vision, representing an incremental improvement over existing NeRF-based methods.
The paper tackles the problem of digitizing real-world objects by estimating geometry, material properties, and environmental lighting from posed images, achieving state-of-the-art relighting quality in about 1 hour of training on a single GPU.
We present a novel approach for digitizing real-world objects by estimating their geometry, material properties, and environmental lighting from a set of posed images with fixed lighting. Our method incorporates into Neural Radiance Field (NeRF) pipelines the split sum approximation used with image-based lighting for real-time physical-based rendering. We propose modeling the scene's lighting with a single scene-specific MLP representing pre-integrated image-based lighting at arbitrary resolutions. We achieve accurate modeling of pre-integrated lighting by exploiting a novel regularizer based on efficient Monte Carlo sampling. Additionally, we propose a new method of supervising self-occlusion predictions by exploiting a similar regularizer based on Monte Carlo sampling. Experimental results demonstrate the efficiency and effectiveness of our approach in estimating scene geometry, material properties, and lighting. Our method is capable of attaining state-of-the-art relighting quality after only ${\sim}1$ hour of training in a single NVIDIA A100 GPU.