Shape and Reflectance Reconstruction in Uncontrolled Environments by Differentiable Rendering
This addresses the challenge of creating photo-realistic virtual scenes from real-world captures for applications like VR or graphics, though it appears incremental as it builds on differentiable rendering.
The paper tackles the problem of reconstructing 3D geometry and reflectance from multi-view photos in uncontrolled settings, achieving superior performance in novel view synthesis compared to state-of-the-art methods.
Simultaneous reconstruction of geometry and reflectance properties in uncontrolled environments remains a challenging problem. In this paper, we propose an efficient method to reconstruct the scene's 3D geometry and reflectance from multi-view photography using conventional hand-held cameras. Our method automatically builds a virtual scene in a differentiable rendering system that roughly matches the real world's scene parameters, optimized by minimizing photometric objectives alternatingly and stochastically. With the optimal scene parameters evaluated, photo-realistic novel views for various viewing angles and distances can then be generated by our approach. We present the results of captured scenes with complex geometry and various reflection types. Our method also shows superior performance compared to state-of-the-art alternatives in novel view synthesis visually and quantitatively.