Caption This, Reason That: VLMs Caught in the Middle
This work provides a detailed cognitive analysis identifying key bottlenecks in VLMs' simultaneous perception and reasoning, offering an effective solution for researchers and developers working on visual AI systems.
The paper analyzed cognitive limitations of Vision-Language Models (VLMs) in visual tasks like spatial reasoning and selective attention, finding that models struggling with direct visual reasoning improved significantly when reasoning over their own generated captions. It demonstrated that targeted fine-tuning on composite visual reasoning tasks substantially improves core cognitive abilities in smaller VLMs, though this doesn't translate to large enhancements on challenging out-of-distribution benchmarks.
Vision-Language Models (VLMs) have shown remarkable progress in visual understanding in recent years. Yet, they still lag behind human capabilities in specific visual tasks such as counting or relational reasoning. To understand the underlying limitations, we adopt methodologies from cognitive science, analyzing VLM performance along core cognitive axes: Perception, Attention, and Memory. Using a suite of tasks targeting these abilities, we evaluate state-of-the-art VLMs, including GPT-4o. Our analysis reveals distinct cognitive profiles: while advanced models approach ceiling performance on some tasks (e.g. category identification), a significant gap persists, particularly in tasks requiring spatial understanding or selective attention. Investigating the source of these failures and potential methods for improvement, we employ a vision-text decoupling analysis, finding that models struggling with direct visual reasoning show marked improvement when reasoning over their own generated text captions. These experiments reveal a strong need for improved VLM Chain-of-Thought (CoT) abilities, even in models that consistently exceed human performance. Furthermore, we demonstrate the potential of targeted fine-tuning on composite visual reasoning tasks and show that fine-tuning smaller VLMs substantially improves core cognitive abilities. While this improvement does not translate to large enhancements on challenging, out-of-distribution benchmarks, we show broadly that VLM performance on our datasets strongly correlates with performance on these other benchmarks. Our work provides a detailed analysis of VLM cognitive strengths and weaknesses and identifies key bottlenecks in simultaneous perception and reasoning while also providing an effective and simple solution.