PBR3DGen: A VLM-guided Mesh Generation with High-quality PBR Texture
This work addresses the problem of realistic 3D rendering for applications in computer graphics and virtual reality, representing an incremental improvement by focusing on spatially varying material properties.
The paper tackles the challenge of generating high-quality physically based rendering (PBR) materials for realistic 3D rendering by introducing PBR3DGen, a two-stage mesh generation method that integrates multi-view PBR material estimation and 3D reconstruction, achieving state-of-the-art results in PBR estimation and mesh generation.
Generating high-quality physically based rendering (PBR) materials is important to achieve realistic rendering in the downstream tasks, yet it remains challenging due to the intertwined effects of materials and lighting. While existing methods have made breakthroughs by incorporating material decomposition in the 3D generation pipeline, they tend to bake highlights into albedo and ignore spatially varying properties of metallicity and roughness. In this work, we present PBR3DGen, a two-stage mesh generation method with high-quality PBR materials that integrates the novel multi-view PBR material estimation model and a 3D PBR mesh reconstruction model. Specifically, PBR3DGen leverages vision language models (VLM) to guide multi-view diffusion, precisely capturing the spatial distribution and inherent attributes of reflective-metalness material. Additionally, we incorporate view-dependent illumination-aware conditions as pixel-aware priors to enhance spatially varying material properties. Furthermore, our reconstruction model reconstructs high-quality mesh with PBR materials. Experimental results demonstrate that PBR3DGen significantly outperforms existing methods, achieving new state-of-the-art results for PBR estimation and mesh generation. More results and visualization can be found on our project page: https://pbr3dgen1218.github.io/.