ShapeCraft: LLM Agents for Structured, Textured and Interactive 3D Modeling
This addresses the need for more practical and accessible 3D modeling tools for artistic workflows, though it appears incremental as it builds on LLM-based agents with a novel representation.
The paper tackled the problem of generating structured, textured, and interactive 3D models from natural language, which existing methods often fail to produce, and demonstrated ShapeCraft's superior performance in creating geometrically accurate and semantically rich 3D assets through qualitative and quantitative experiments.
3D generation from natural language offers significant potential to reduce expert manual modeling efforts and enhance accessibility to 3D assets. However, existing methods often yield unstructured meshes and exhibit poor interactivity, making them impractical for artistic workflows. To address these limitations, we represent 3D assets as shape programs and introduce ShapeCraft, a novel multi-agent framework for text-to-3D generation. At its core, we propose a Graph-based Procedural Shape (GPS) representation that decomposes complex natural language into a structured graph of sub-tasks, thereby facilitating accurate LLM comprehension and interpretation of spatial relationships and semantic shape details. Specifically, LLM agents hierarchically parse user input to initialize GPS, then iteratively refine procedural modeling and painting to produce structured, textured, and interactive 3D assets. Qualitative and quantitative experiments demonstrate ShapeCraft's superior performance in generating geometrically accurate and semantically rich 3D assets compared to existing LLM-based agents. We further show the versatility of ShapeCraft through examples of animated and user-customized editing, highlighting its potential for broader interactive applications.