ControlMat: A Controlled Generative Approach to Material Capture
This addresses the problem of 3D content creation democratization by enabling material capture from images, though it appears incremental in adapting existing generative techniques.
The paper tackles material reconstruction from a single photograph by formulating it as a controlled synthesis problem, using a diffusion model called ControlMat to generate plausible, tileable, high-resolution materials, and shows it outperforms recent methods.
Material reconstruction from a photograph is a key component of 3D content creation democratization. We propose to formulate this ill-posed problem as a controlled synthesis one, leveraging the recent progress in generative deep networks. We present ControlMat, a method which, given a single photograph with uncontrolled illumination as input, conditions a diffusion model to generate plausible, tileable, high-resolution physically-based digital materials. We carefully analyze the behavior of diffusion models for multi-channel outputs, adapt the sampling process to fuse multi-scale information and introduce rolled diffusion to enable both tileability and patched diffusion for high-resolution outputs. Our generative approach further permits exploration of a variety of materials which could correspond to the input image, mitigating the unknown lighting conditions. We show that our approach outperforms recent inference and latent-space-optimization methods, and carefully validate our diffusion process design choices. Supplemental materials and additional details are available at: https://gvecchio.com/controlmat/.