Alchemist: Parametric Control of Material Properties with Diffusion Models
This addresses a lack of datasets for material control in computer vision, offering a method for realistic image editing, though it is incremental as it builds on existing diffusion models.
The paper tackles the problem of controlling material attributes like roughness and metallic in real images by fine-tuning a pre-trained text-to-image model on a synthetic dataset, enabling editing while preserving other attributes and showing potential for material-edited NeRFs.
We propose a method to control material attributes of objects like roughness, metallic, albedo, and transparency in real images. Our method capitalizes on the generative prior of text-to-image models known for photorealism, employing a scalar value and instructions to alter low-level material properties. Addressing the lack of datasets with controlled material attributes, we generated an object-centric synthetic dataset with physically-based materials. Fine-tuning a modified pre-trained text-to-image model on this synthetic dataset enables us to edit material properties in real-world images while preserving all other attributes. We show the potential application of our model to material edited NeRFs.