PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes
This work addresses the need for photorealistic scene reconstruction in computer vision and graphics, enabling applications like virtual reality and digital content creation, though it builds on existing reconstruction methods.
The authors tackled the problem of creating photorealistic digital twins of indoor scenes by transferring materials and lighting from input images to aligned CAD geometry, achieving high-quality, fully relightable 3D scenes that can be re-rendered under arbitrary conditions.
Most indoor 3D scene reconstruction methods focus on recovering 3D geometry and scene layout. In this work, we go beyond this to propose PhotoScene, a framework that takes input image(s) of a scene along with approximately aligned CAD geometry (either reconstructed automatically or manually specified) and builds a photorealistic digital twin with high-quality materials and similar lighting. We model scene materials using procedural material graphs; such graphs represent photorealistic and resolution-independent materials. We optimize the parameters of these graphs and their texture scale and rotation, as well as the scene lighting to best match the input image via a differentiable rendering layer. We evaluate our technique on objects and layout reconstructions from ScanNet, SUN RGB-D and stock photographs, and demonstrate that our method reconstructs high-quality, fully relightable 3D scenes that can be re-rendered under arbitrary viewpoints, zooms and lighting.