Diffusion-Based Material Regularization for Physics-Based Inverse Rendering
For computer graphics and vision researchers, this work bridges physics-based and data-driven approaches to produce high-quality, relightable 3D assets from multi-view images.
The paper addresses the underconstrained nature of physics-based inverse rendering by introducing a diffusion-based regularization loss that uses diffusion model predictions as a similarity kernel, enabling joint reconstruction of geometry, materials, and illumination. The method significantly outperforms state-of-the-art baselines on Synthetic4Relight, Stanford-ORB, and DTC-Synthetic datasets in both reconstruction accuracy and relighting quality.
Reconstructing physics-based 3D assets -- geometry, materials, and illumination -- from multi-view images is a core problem in computer graphics and vision, and a prerequisite for realistic relighting and editing. Physics-based inverse rendering offers an accurate image-formation model, but is severely underconstrained: without strong priors, illumination is baked into materials, and reconstructions generalize poorly to novel views and lighting. Data-driven diffusion models, in contrast, predict visually plausible materials, yet their predictions rarely satisfy the rendering equation and are not directly usable for physics-based rendering. We bridge these two paradigms rather than replacing either. Our key idea is to treat the predictions of a state-of-the-art diffusion model not as target material values but as a similarity kernel for optimization: we introduce a regularization loss that penalizes deviations in the optimized material over surface regions where the diffusion predictions are near-constant, while leaving the optimization free to match the input images. Built on this regularizer, our end-to-end pipeline jointly reconstructs geometry, materials, and illumination, yielding high-quality assets that drop into standard rendering pipelines and relight faithfully. On the Synthetic4Relight, Stanford-ORB, and DTC-Synthetic datasets, our method significantly outperforms state-of-the-art baselines in both reconstruction accuracy and relighting quality.