Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation
This work addresses a key challenge in virtual content creation, gaming, and embodied AI by improving scene generation realism, though it is incremental as it builds on existing LLM and vision methods.
The paper tackles the problem of generating interactive 3D scenes from text, which is challenging due to limited dataset diversity and unrealistic spatial layouts from language models, and introduces Scenethesis, a training-free framework that integrates language and vision to produce diverse, realistic, and physically plausible scenes.
Synthesizing interactive 3D scenes from text is essential for gaming, virtual reality, and embodied AI. However, existing methods face several challenges. Learning-based approaches depend on small-scale indoor datasets, limiting the scene diversity and layout complexity. While large language models (LLMs) can leverage diverse text-domain knowledge, they struggle with spatial realism, often producing unnatural object placements that fail to respect common sense. Our key insight is that vision perception can bridge this gap by providing realistic spatial guidance that LLMs lack. To this end, we introduce Scenethesis, a training-free agentic framework that integrates LLM-based scene planning with vision-guided layout refinement. Given a text prompt, Scenethesis first employs an LLM to draft a coarse layout. A vision module then refines it by generating an image guidance and extracting scene structure to capture inter-object relations. Next, an optimization module iteratively enforces accurate pose alignment and physical plausibility, preventing artifacts like object penetration and instability. Finally, a judge module verifies spatial coherence. Comprehensive experiments show that Scenethesis generates diverse, realistic, and physically plausible 3D interactive scenes, making it valuable for virtual content creation, simulation environments, and embodied AI research.