LLM-Guided Material Inference for 3D Point Clouds
This addresses the gap in material understanding for 3D shape analysis, benefiting applications in computer vision and graphics, though it is incremental as it builds on existing LLM capabilities for a new task.
The paper tackles the problem of inferring material composition from 3D point clouds, which existing datasets and models often overlook, by introducing a two-stage LLM-based method that achieves high semantic and material plausibility across 1,000 shapes from Fusion/ABS and ShapeNet.
Most existing 3D shape datasets and models focus solely on geometry, overlooking the material properties that determine how objects appear. We introduce a two-stage large language model (LLM) based method for inferring material composition directly from 3D point clouds with coarse segmentations. Our key insight is to decouple reasoning about what an object is from what it is made of. In the first stage, an LLM predicts the object's semantic; in the second stage, it assigns plausible materials to each geometric segment, conditioned on the inferred semantics. Both stages operate in a zero-shot manner, without task-specific training. Because existing datasets lack reliable material annotations, we evaluate our method using an LLM-as-a-Judge implemented in DeepEval. Across 1,000 shapes from Fusion/ABS and ShapeNet, our method achieves high semantic and material plausibility. These results demonstrate that language models can serve as general-purpose priors for bridging geometric reasoning and material understanding in 3D data.