DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer
This work addresses the challenge of inverse graphics for computer vision and graphics applications, offering a more realistic and efficient approach for tasks like material editing and relighting, though it is incremental in improving over previous differentiable renderers.
The paper tackles the problem of predicting intrinsic object properties like lighting and material from a single image by introducing DIB-R++, a hybrid differentiable renderer that combines rasterization and ray-tracing to handle photorealistic effects such as specular reflections. It achieves superior material and lighting disentanglement on synthetic and real data compared to existing rasterization-based methods.
We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiable renderers. Many previous learning-based approaches for inverse graphics adopt rasterization-based renderers and assume naive lighting and material models, which often fail to account for non-Lambertian, specular reflections commonly observed in the wild. In this work, we propose DIBR++, a hybrid differentiable renderer which supports these photorealistic effects by combining rasterization and ray-tracing, taking the advantage of their respective strengths -- speed and realism. Our renderer incorporates environmental lighting and spatially-varying material models to efficiently approximate light transport, either through direct estimation or via spherical basis functions. Compared to more advanced physics-based differentiable renderers leveraging path tracing, DIBR++ is highly performant due to its compact and expressive shading model, which enables easy integration with learning frameworks for geometry, reflectance and lighting prediction from a single image without requiring any ground-truth. We experimentally demonstrate that our approach achieves superior material and lighting disentanglement on synthetic and real data compared to existing rasterization-based approaches and showcase several artistic applications including material editing and relighting.