SPIDR: SDF-based Neural Point Fields for Illumination and Deformation
This work addresses the challenge for users in computer graphics and vision who need to edit and relight scenes after deformation, representing an incremental improvement over existing NeRF-based methods.
The authors tackled the problem of enabling explicit manipulation of shape, reflectance, and illumination in neural radiance fields (NeRFs) for 3D reconstruction, introducing SPIDR to achieve higher quality object surfaces and more accurate scene illumination updates after deformation.
Neural radiance fields (NeRFs) have recently emerged as a promising approach for 3D reconstruction and novel view synthesis. However, NeRF-based methods encode shape, reflectance, and illumination implicitly and this makes it challenging for users to manipulate these properties in the rendered images explicitly. Existing approaches only enable limited editing of the scene and deformation of the geometry. Furthermore, no existing work enables accurate scene illumination after object deformation. In this work, we introduce SPIDR, a new hybrid neural SDF representation. SPIDR combines point cloud and neural implicit representations to enable the reconstruction of higher quality object surfaces for geometry deformation and lighting estimation. meshes and surfaces for object deformation and lighting estimation. To more accurately capture environment illumination for scene relighting, we propose a novel neural implicit model to learn environment light. To enable more accurate illumination updates after deformation, we use the shadow mapping technique to approximate the light visibility updates caused by geometry editing. We demonstrate the effectiveness of SPIDR in enabling high quality geometry editing with more accurate updates to the illumination of the scene.