I Know About "Up"! Enhancing Spatial Reasoning in Visual Language Models Through 3D Reconstruction
This addresses a specific limitation in VLMs for tasks requiring spatial reasoning, representing an incremental improvement through hybrid methods.
The paper tackles the problem of inadequate visual spatial reasoning in Visual Language Models (VLMs), which struggle with basic tasks like distinguishing left from right, by proposing the ZeroVLM model that uses 3D reconstruction and prompting mechanisms, resulting in up to 19.48% accuracy improvement on four datasets.
Visual Language Models (VLMs) are essential for various tasks, particularly visual reasoning tasks, due to their robust multi-modal information integration, visual reasoning capabilities, and contextual awareness. However, existing \VLMs{}' visual spatial reasoning capabilities are often inadequate, struggling even with basic tasks such as distinguishing left from right. To address this, we propose the \ours{} model, designed to enhance the visual spatial reasoning abilities of VLMS. ZeroVLM employs Zero-1-to-3, a 3D reconstruction model for obtaining different views of the input images and incorporates a prompting mechanism to further improve visual spatial reasoning. Experimental results on four visual spatial reasoning datasets show that our \ours{} achieves up to 19.48% accuracy improvement, which indicates the effectiveness of the 3D reconstruction and prompting mechanisms of our ZeroVLM.