MultiMat: Multimodal Program Synthesis for Procedural Materials using Large Multimodal Models
This work addresses the problem of simplifying procedural material graph creation for computer graphics professionals, though it is incremental as it builds on existing neural program synthesis methods by adding multimodal capabilities.
The paper tackles the challenge of creating procedural material node graphs, which are essential for computer graphics but require professional training, by introducing MultiMat, a multimodal program synthesis framework that uses large multimodal models to process both visual and textual representations. The results show that MultiMat is more efficient and achieves higher visual quality and fidelity than text-only baselines, establishing new state-of-the-art performance.
Material node graphs are programs that generate the 2D channels of procedural materials, including geometry such as roughness and displacement maps, and reflectance such as albedo and conductivity maps. They are essential in computer graphics for representing the appearance of virtual 3D objects parametrically and at arbitrary resolution. In particular, their directed acyclic graph structures and intermediate states provide an intuitive understanding and workflow for interactive appearance modeling. Creating such graphs is a challenging task and typically requires professional training. While recent neural program synthesis approaches attempt to simplify this process, they solely represent graphs as textual programs, failing to capture the inherently visual-spatial nature of node graphs that makes them accessible to humans. To address this gap, we present MultiMat, a multimodal program synthesis framework that leverages large multimodal models to process both visual and textual graph representations for improved generation of procedural material graphs. We train our models on a new dataset of production-quality procedural materials and combine them with a constrained tree search inference algorithm that ensures syntactic validity while efficiently navigating the program space. Our experimental results show that our multimodal program synthesis method is more efficient in both unconditional and conditional graph synthesis with higher visual quality and fidelity than text-only baselines, establishing new state-of-the-art performance.