VideoMatGen: PBR Materials through Joint Generative Modeling
This addresses the problem of efficient material generation for 3D content creators, though it appears incremental as it builds on existing diffusion and VAE techniques.
The paper tackles generating physically-based materials for 3D shapes by using a video diffusion transformer conditioned on geometry and text, resulting in high-quality materials compatible with content creation tools.
We present a method for generating physically-based materials for 3D shapes based on a video diffusion transformer architecture. Our method is conditioned on input geometry and a text description, and jointly models multiple material properties (base color, roughness, metallicity, height map) to form physically plausible materials. We further introduce a custom variational auto-encoder which encodes multiple material modalities into a compact latent space, which enables joint generation of multiple modalities without increasing the number of tokens. Our pipeline generates high-quality materials for 3D shapes given a text prompt, compatible with common content creation tools.