Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs
This addresses a core limitation in LVLMs for tasks like counting and spatial reasoning, offering a general approach to enhance compositional and spatial reasoning, though it is incremental as it builds on existing models with input modifications.
The paper tackles the binding problem in Large Vision-Language Models (LVLMs), which limits visual reasoning by failing to associate perceptual features with correct visual referents, and introduces VISER, a method that augments visual inputs with spatial structures and textual prompts to improve performance. Results show improvements of 25.0% on visual search, 26.8% on counting, and 9.5% on spatial relationship tasks with GPT-4o, and a reduction of 0.32 in edit distance error for scene description.
Despite progress in Large Vision-Language Models (LVLMs), their capacity for visual reasoning is often limited by the binding problem: the failure to reliably associate perceptual features with their correct visual referents. This limitation underlies persistent errors in tasks such as counting, visual search, scene description, and spatial relationship understanding. A key factor is that current LVLMs process visual features largely in parallel, lacking mechanisms for spatially grounded, serial attention. This paper introduces Visual Input Structure for Enhanced Reasoning (VISER), a simple, effective method that augments visual inputs with low-level spatial structures and pairs them with a textual prompt that encourages sequential, spatially-aware parsing. We empirically demonstrate substantial performance improvements across core visual reasoning tasks, using only a single-query inference. Specifically, VISER improves GPT-4o performance on visual search, counting, and spatial relationship tasks by 25.0%, 26.8%, and 9.5%, respectively, and reduces edit distance error in scene description by 0.32 on 2D datasets. Furthermore, we find that the visual modification is essential for these gains; purely textual strategies, including Chain-of-Thought prompting, are insufficient and can even degrade performance. VISER underscores the importance of visual input design over purely linguistically based reasoning strategies and suggests that visual structuring is a powerful and general approach for enhancing compositional and spatial reasoning in LVLMs.