VideoMat: Extracting PBR Materials from Video Diffusion Models
This addresses the challenge of material creation for 3D content generation, offering a novel pipeline that integrates multiple existing techniques for domain-specific applications in computer graphics.
The paper tackles the problem of generating high-quality physically-based rendering (PBR) materials for 3D models from text prompts or single images by leveraging finetuned video diffusion models, intrinsic decomposition, and differentiable rendering to produce coherent and compatible materials.
We leverage finetuned video diffusion models, intrinsic decomposition of videos, and physically-based differentiable rendering to generate high quality materials for 3D models given a text prompt or a single image. We condition a video diffusion model to respect the input geometry and lighting condition. This model produces multiple views of a given 3D model with coherent material properties. Secondly, we use a recent model to extract intrinsics (base color, roughness, metallic) from the generated video. Finally, we use the intrinsics alongside the generated video in a differentiable path tracer to robustly extract PBR materials directly compatible with common content creation tools.