VRIQ: Benchmarking and Analyzing Visual-Reasoning IQ of VLMs
This work addresses the reliability of VLMs for nonverbal reasoning, providing a diagnostic benchmark for researchers, but it is incremental as it builds on existing evaluation methods.
The authors tackled the problem of assessing visual reasoning in Vision Language Models (VLMs) by introducing the VRIQ benchmark, finding that performance is near random (28% accuracy) on abstract puzzles and weak (45% accuracy) on natural tasks, with most failures attributed to perception issues.
Recent progress in Vision Language Models (VLMs) has raised the question of whether they can reliably perform nonverbal reasoning. To this end, we introduce VRIQ (Visual Reasoning IQ), a novel benchmark designed to assess and analyze the visual reasoning ability of VLMs. We evaluate models on two sets of tasks: abstract puzzle-style and natural-image reasoning tasks. We find that on abstract puzzles, performance remains near random with an average accuracy of around 28%, while natural tasks yield better but still weak results with 45% accuracy. We also find that tool-augmented reasoning demonstrates only modest improvements. To uncover the source of this weakness, we introduce diagnostic probes targeting perception and reasoning. Our analysis demonstrates that around 56% of failures arise from perception alone, 43% from both perception and reasoning, and only a mere 1% from reasoning alone. This motivates us to design fine-grained diagnostic probe questions targeting specific perception categories (e.g., shape, count, position, 3D/depth), revealing that certain categories cause more failures than others. Our benchmark and analysis establish that current VLMs, even with visual reasoning tools, remain unreliable abstract reasoners, mostly due to perception limitations, and offer a principled basis for improving visual reasoning in multimodal systems.