Learning-based Inverse Rendering of Complex Indoor Scenes with Differentiable Monte Carlo Raytracing
This work addresses the ill-posed problem of inverse rendering for indoor scenes, which has applications in object insertion and material editing, though it appears incremental with novel components like out-of-view lighting.
The authors tackled the challenging problem of inverse rendering for complex indoor scenes by developing an end-to-end learning-based framework that recovers geometry, lighting, and materials from a single image, achieving superior quality compared to state-of-the-art baselines.
Indoor scenes typically exhibit complex, spatially-varying appearance from global illumination, making inverse rendering a challenging ill-posed problem. This work presents an end-to-end, learning-based inverse rendering framework incorporating differentiable Monte Carlo raytracing with importance sampling. The framework takes a single image as input to jointly recover the underlying geometry, spatially-varying lighting, and photorealistic materials. Specifically, we introduce a physically-based differentiable rendering layer with screen-space ray tracing, resulting in more realistic specular reflections that match the input photo. In addition, we create a large-scale, photorealistic indoor scene dataset with significantly richer details like complex furniture and dedicated decorations. Further, we design a novel out-of-view lighting network with uncertainty-aware refinement leveraging hypernetwork-based neural radiance fields to predict lighting outside the view of the input photo. Through extensive evaluations on common benchmark datasets, we demonstrate superior inverse rendering quality of our method compared to state-of-the-art baselines, enabling various applications such as complex object insertion and material editing with high fidelity. Code and data will be made available at \url{https://jingsenzhu.github.io/invrend}.