Extracting Neural Materials from Multi-view Images
For computer graphics practitioners, it enables high-quality material extraction from images, addressing a key bottleneck in neural material acquisition.
NeuMatEx extracts spatially varying neural materials from multi-view images using a Large Material Reconstruction Model (LMRM) to initialize and constrain optimization via inverse path tracing, achieving better visual quality and material decomposition than PBR-based methods.
Neural materials can represent complex specular reflections and scattering effects in a compact, universal basis. However, acquiring and authoring such materials remains challenging. We present NeuMatEx, a differentiable inverse rendering method for extracting spatially varying neural materials from images. The nonlinear structure of neural material latent spaces makes optimization with naive inverse rendering infeasible. To address this, we train a Large Material Reconstruction Model (LMRM) that directly predicts initialbase color, neural material latents, and aleatoric uncertainty guides from images. This material prior provides a good initialization and better constrains our subsequent optimization using inverse path tracing. The predicted uncertainty further helps by anchoring high-confidence regions more tightly to the LMRM prediction, preventing lighting and complex specular effects from being baked into materials. Experiments on synthetic and real assets show that NeuMatEx extracts complex materials with better visual quality and material decomposition than PBR-based methods.