A Cognitive Paradigm Approach to Probe the Perception-Reasoning Interface in VLMs
This provides a diagnostic tool for researchers to identify perception bottlenecks in VLMs, offering a promising direction for improving robust visual intelligence, though it is incremental in advancing evaluation methods rather than solving the core problem.
The paper tackles the challenge of understanding how Vision-Language Models integrate visual perception with abstract reasoning by introducing a cognitive-inspired evaluation framework using tasks like Bongard Problems and Winoground, and demonstrates that decoupling perception from reasoning via componential analysis achieves new state-of-the-art performance on benchmarks such as Bongard-OpenWorld, Bongard-HOI, and Winoground.
A fundamental challenge in artificial intelligence involves understanding the cognitive mechanisms underlying visual reasoning in sophisticated models like Vision-Language Models (VLMs). How do these models integrate visual perception with abstract thought, especially when reasoning across multiple images or requiring fine-grained compositional understanding? Drawing inspiration from cognitive science, this paper introduces a structured evaluation framework using diverse visual reasoning tasks-Bongard Problems (BPs) and Winoground-to dissect the perception-reasoning interface in VLMs. We propose three distinct evaluation paradigms, mirroring human problem-solving strategies: Direct Visual Rule Learning (DVRL; holistic processing), Deductive Rule Learning (DRL; rule extraction and application), and Componential Analysis (CA; analytical decomposition via task-agnostic textual descriptions). These paradigms systematically vary cognitive load and probe processing stages. Notably, CA enables multi-image reasoning evaluation even for single-image architectures and isolates reasoning from perception by operating on textual descriptions. Applying this framework, we demonstrate that CA, leveraging powerful language models for reasoning over rich, independently generated descriptions, achieves new state-of-the-art (SOTA) performance on challenging benchmarks including Bongard-OpenWorld, Bongard-HOI, and Winoground. Ablation studies confirm reasoning improves significantly when perceptual challenges are mitigated, revealing a critical perception bottleneck. Our framework provides a valuable diagnostic tool and suggests that decoupling perception (via rich, task-agnostic description) from reasoning is a promising direction for robust and general visual intelligence.