DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models
This addresses the challenge of rendering in computer vision and graphics without needing precise scene representations, enabling practical applications like relighting and material editing from a single video input.
The paper tackles the problem of inverse and forward rendering by introducing DiffusionRenderer, a neural approach that uses video diffusion models to estimate G-buffers from real-world videos and generate photorealistic images without explicit light transport simulation, consistently outperforming state-of-the-art methods.
Understanding and modeling lighting effects are fundamental tasks in computer vision and graphics. Classic physically-based rendering (PBR) accurately simulates the light transport, but relies on precise scene representations--explicit 3D geometry, high-quality material properties, and lighting conditions--that are often impractical to obtain in real-world scenarios. Therefore, we introduce DiffusionRenderer, a neural approach that addresses the dual problem of inverse and forward rendering within a holistic framework. Leveraging powerful video diffusion model priors, the inverse rendering model accurately estimates G-buffers from real-world videos, providing an interface for image editing tasks, and training data for the rendering model. Conversely, our rendering model generates photorealistic images from G-buffers without explicit light transport simulation. Experiments demonstrate that DiffusionRenderer effectively approximates inverse and forwards rendering, consistently outperforming the state-of-the-art. Our model enables practical applications from a single video input--including relighting, material editing, and realistic object insertion.