Layout-your-3D: Controllable and Precise 3D Generation with 2D Blueprint
This addresses the challenge of generating plausible 3D assets with precise control for applications like 3D editing and object insertion, representing an incremental improvement over existing text-to-3D methods.
The paper tackles the problem of controllable and compositional 3D generation from text prompts by using 2D layouts as blueprints to improve object interactions and reduce optimization time, resulting in more reasonable and visually appealing 3D assets with significantly reduced time per prompt.
We present Layout-Your-3D, a framework that allows controllable and compositional 3D generation from text prompts. Existing text-to-3D methods often struggle to generate assets with plausible object interactions or require tedious optimization processes. To address these challenges, our approach leverages 2D layouts as a blueprint to facilitate precise and plausible control over 3D generation. Starting with a 2D layout provided by a user or generated from a text description, we first create a coarse 3D scene using a carefully designed initialization process based on efficient reconstruction models. To enforce coherent global 3D layouts and enhance the quality of instance appearances, we propose a collision-aware layout optimization process followed by instance-wise refinement. Experimental results demonstrate that Layout-Your-3D yields more reasonable and visually appealing compositional 3D assets while significantly reducing the time required for each prompt. Additionally, Layout-Your-3D can be easily applicable to downstream tasks, such as 3D editing and object insertion. Our project page is available at:https://colezwhy.github.io/layoutyour3d/