Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas
This work addresses a specific bottleneck in VLMs for spatial reasoning tasks, offering an incremental improvement with practical applications in AI vision-language understanding.
The paper tackled the problem of spatial reasoning difficulties in Vision Language Models (VLMs) by analyzing attention mechanisms and proposed a training-free decoding method, ADAPTVIS, which achieved up to a 50-point improvement on benchmarks like WhatsUp and VSR.
Large Vision Language Models (VLMs) have long struggled with spatial reasoning tasks. Surprisingly, even simple spatial reasoning tasks, such as recognizing "under" or "behind" relationships between only two objects, pose significant challenges for current VLMs. In this work, we study the spatial reasoning challenge from the lens of mechanistic interpretability, diving into the model's internal states to examine the interactions between image and text tokens. By tracing attention distribution over the image through out intermediate layers, we observe that successful spatial reasoning correlates strongly with the model's ability to align its attention distribution with actual object locations, particularly differing between familiar and unfamiliar spatial relationships. Motivated by these findings, we propose ADAPTVIS based on inference-time confidence scores to sharpen the attention on highly relevant regions when confident, while smoothing and broadening the attention window to consider a wider context when confidence is lower. This training-free decoding method shows significant improvement (e.g., up to a 50 absolute point improvement) on spatial reasoning benchmarks such as WhatsUp and VSR with negligible cost. We make code and data publicly available for research purposes at https://github.com/shiqichen17/AdaptVis.