LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models
This addresses a fundamental challenge in AI for intuitive 3D object manipulation, offering a novel approach that improves over existing methods, though it appears incremental in combining VLMs with optimization techniques.
The paper tackles the problem of 3D spatial reasoning for arranging objects according to language instructions in dense, constrained environments, introducing LayoutVLM, which uses vision-language models and differentiable optimization to produce physically plausible layouts better aligned with semantic intent.
Spatial reasoning is a fundamental aspect of human cognition, enabling intuitive understanding and manipulation of objects in three-dimensional space. While foundation models demonstrate remarkable performance on some benchmarks, they still struggle with 3D reasoning tasks like arranging objects in space according to open-ended language instructions, particularly in dense and physically constrained environments. We introduce LayoutVLM, a framework and scene layout representation that exploits the semantic knowledge of Vision-Language Models (VLMs) and supports differentiable optimization to ensure physical plausibility. LayoutVLM employs VLMs to generate two mutually reinforcing representations from visually marked images, and a self-consistent decoding process to improve VLMs spatial planning. Our experiments show that LayoutVLM addresses the limitations of existing LLM and constraint-based approaches, producing physically plausible 3D layouts better aligned with the semantic intent of input language instructions. We also demonstrate that fine-tuning VLMs with the proposed scene layout representation extracted from existing scene datasets can improve their reasoning performance.