VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs
This work identifies a critical limitation in current VLMs for applications requiring complex visual reasoning, such as robotics or medical imaging, though it is incremental as it builds on prior evidence of perceptual weaknesses.
The paper evaluated leading visual language models (VLMs) on nonlocal visual reasoning tasks, such as comparative perception and visual search, and found that flagship models like Gemini 2.5 Pro and GPT-o4-mini failed these tests, performing barely above random accuracy on tasks trivial for humans.
Visual Language Models (VLMs) excel at complex visual tasks such as VQA and chart understanding, yet recent work suggests they struggle with simple perceptual tests. We present an evaluation that tests vision-language models' capacity for nonlocal visual reasoning -- reasoning that requires chaining evidence collected from multiple, possibly distant, regions of an image. We isolate three distinct forms of non-local vision: comparative perception, which demands holding two images in working memory and comparing them; saccadic search, which requires making discrete, evidence-driven jumps to locate successive targets; and smooth visual search, which involves searching smoothly along a continuous contour. Flagship models (e.g., Gemini 2.5 Pro, Claude Vision 3.7, GPT-o4-mini), even those that perform well on prior primitive-vision benchmarks, fail these tests and barely exceed random accuracy on two variants of our tasks that are trivial for humans. Our structured evaluation suite allows us to test if VLMs can perform similar visual algorithms to humans. Our findings show that despite gains in raw visual acuity, current models lack core visual reasoning capabilities.