MatCLIP: Light- and Shape-Insensitive Assignment of PBR Material Models
This addresses the problem of material assignment in computer graphics for applications like 3D modeling and rendering, representing a strong specific gain rather than a foundational advancement.
The paper tackles the challenge of assigning realistic materials to 3D models by proposing MatCLIP, a method that extracts shape- and lighting-insensitive descriptors for PBR materials to assign textures from images, achieving a top-1 classification accuracy of 76.6% and outperforming state-of-the-art methods by over 15 percentage points.
Assigning realistic materials to 3D models remains a significant challenge in computer graphics. We propose MatCLIP, a novel method that extracts shape- and lighting-insensitive descriptors of Physically Based Rendering (PBR) materials to assign plausible textures to 3D objects based on images, such as the output of Latent Diffusion Models (LDMs) or photographs. Matching PBR materials to static images is challenging because the PBR representation captures the dynamic appearance of materials under varying viewing angles, shapes, and lighting conditions. By extending an Alpha-CLIP-based model on material renderings across diverse shapes and lighting, and encoding multiple viewing conditions for PBR materials, our approach generates descriptors that bridge the domains of PBR representations with photographs or renderings, including LDM outputs. This enables consistent material assignments without requiring explicit knowledge of material relationships between different parts of an object. MatCLIP achieves a top-1 classification accuracy of 76.6%, outperforming state-of-the-art methods such as PhotoShape and MatAtlas by over 15 percentage points on publicly available datasets. Our method can be used to construct material assignments for 3D shape datasets such as ShapeNet, 3DCoMPaT++, and Objaverse. All code and data will be released.