S$^3$-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint
This addresses the challenge of 3D scene reconstruction from limited views for applications in computer vision and graphics, representing an incremental advance over existing single-view methods.
The paper tackles the problem of reconstructing 3D geometry and reflectance from single-view images under varying lighting, using shading and shadow cues to recover both visible and invisible parts of a scene, with results showing robustness to depth discontinuities and support for novel-view synthesis and relighting.
In this paper, we address the "dual problem" of multi-view scene reconstruction in which we utilize single-view images captured under different point lights to learn a neural scene representation. Different from existing single-view methods which can only recover a 2.5D scene representation (i.e., a normal / depth map for the visible surface), our method learns a neural reflectance field to represent the 3D geometry and BRDFs of a scene. Instead of relying on multi-view photo-consistency, our method exploits two information-rich monocular cues, namely shading and shadow, to infer scene geometry. Experiments on multiple challenging datasets show that our method is capable of recovering 3D geometry, including both visible and invisible parts, of a scene from single-view images. Thanks to the neural reflectance field representation, our method is robust to depth discontinuities. It supports applications like novel-view synthesis and relighting. Our code and model can be found at https://ywq.github.io/s3nerf.