FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placement
This work addresses the problem of generating realistic 3D scenes for applications like virtual environments or robotics, though it appears incremental as it refines existing MLLMs with geometric techniques.
The paper tackles the challenge of 3D object placement in scene generation by combining multimodal large language models (MLLMs) with geometric reasoning to address their limited grounding in 3D geometry, resulting in a method that surpasses prior work in placement quality for complex scenes.
Scene generation with 3D assets presents a complex challenge, requiring both high-level semantic understanding and low-level geometric reasoning. While Multimodal Large Language Models (MLLMs) excel at semantic tasks, their application to 3D scene generation is hindered by their limited grounding on 3D geometry. In this paper, we investigate how to best work with MLLMs in an object placement task. Towards this goal, we introduce a novel framework, FirePlace, that applies existing MLLMs in (1) 3D geometric reasoning and the extraction of relevant geometric details from the 3D scene, (2) constructing and solving geometric constraints on the extracted low-level geometry, and (3) pruning for final placements that conform to common sense. By combining geometric reasoning with real-world understanding of MLLMs, our method can propose object placements that satisfy both geometric constraints as well as high-level semantic common-sense considerations. Our experiments show that these capabilities allow our method to place objects more effectively in complex scenes with intricate geometry, surpassing the quality of prior work.