Relighting Scenes with Object Insertions in Neural Radiance Fields
This work addresses the limited application scenarios in AR by enabling more immersive object insertions with relighting, though it is incremental as it builds on existing NeRF methods.
The paper tackles the problem of inserting virtual objects into real scenes with realistic relighting in augmented reality, proposing a NeRF-based pipeline that enables novel view synthesis and supports physical interactions like shadow casting, achieving realistic effects in experiments.
The insertion of objects into a scene and relighting are commonly utilized applications in augmented reality (AR). Previous methods focused on inserting virtual objects using CAD models or real objects from single-view images, resulting in highly limited AR application scenarios. We propose a novel NeRF-based pipeline for inserting object NeRFs into scene NeRFs, enabling novel view synthesis and realistic relighting, supporting physical interactions like casting shadows onto each other, from two sets of images depicting the object and scene. The lighting environment is in a hybrid representation of Spherical Harmonics and Spherical Gaussians, representing both high- and low-frequency lighting components very well, and supporting non-Lambertian surfaces. Specifically, we leverage the benefits of volume rendering and introduce an innovative approach for efficient shadow rendering by comparing the depth maps between the camera view and the light source view and generating vivid soft shadows. The proposed method achieves realistic relighting effects in extensive experimental evaluations.