MATLABER: Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR
This work addresses the challenge of realistic material generation for text-to-3D applications, enabling downstream tasks like relighting and editing, but it is incremental as it builds on existing diffusion models.
The paper tackles the problem of generating high-fidelity object materials in text-to-3D generation by proposing MATLABER, which uses a latent BRDF auto-encoder trained on real-world data to predict material embeddings, resulting in superior realism and coherence compared to existing methods.
Based on powerful text-to-image diffusion models, text-to-3D generation has made significant progress in generating compelling geometry and appearance. However, existing methods still struggle to recover high-fidelity object materials, either only considering Lambertian reflectance, or failing to disentangle BRDF materials from the environment lights. In this work, we propose Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR (\textbf{MATLABER}) that leverages a novel latent BRDF auto-encoder for material generation. We train this auto-encoder with large-scale real-world BRDF collections and ensure the smoothness of its latent space, which implicitly acts as a natural distribution of materials. During appearance modeling in text-to-3D generation, the latent BRDF embeddings, rather than BRDF parameters, are predicted via a material network. Through exhaustive experiments, our approach demonstrates the superiority over existing ones in generating realistic and coherent object materials. Moreover, high-quality materials naturally enable multiple downstream tasks such as relighting and material editing. Code and model will be publicly available at \url{https://sheldontsui.github.io/projects/Matlaber}.