Jigsaw-Puzzles: From Seeing to Understanding to Reasoning in Vision-Language Models
This work addresses the need for better benchmarks to assess spatial reasoning in vision-language models, which is crucial for advancing AI capabilities in perception and interaction, though it is incremental as it focuses on evaluation rather than model improvement.
The authors tackled the problem of evaluating spatial reasoning in vision-language models by introducing the Jigsaw-Puzzles benchmark with 1,100 images and five tasks, finding that even the strongest model, Gemini-2.5-Pro, achieved only 77.14% overall accuracy and 30.00% on a specific task, far below human performance exceeding 90%.
Spatial reasoning is a core component of human cognition, enabling individuals to perceive, comprehend, and interact with the physical world. It relies on a nuanced understanding of spatial structures and inter-object relationships, serving as the foundation for complex reasoning and decision-making. To investigate whether current vision-language models (VLMs) exhibit similar capability, we introduce Jigsaw-Puzzles, a novel benchmark consisting of 1,100 carefully curated real-world images with high spatial complexity. Based on this dataset, we design five tasks to rigorously evaluate VLMs' spatial perception, structural understanding, and reasoning capabilities, while deliberately minimizing reliance on domain-specific knowledge to better isolate and assess the general spatial reasoning capability. We conduct a comprehensive evaluation across 24 state-of-the-art VLMs. The results show that even the strongest model, Gemini-2.5-Pro, achieves only 77.14% overall accuracy and performs particularly poorly on the Order Generation task, with only 30.00% accuracy, far below the performance exceeding 90% achieved by human participants. This persistent gap underscores the need for continued progress, positioning Jigsaw-Puzzles as a challenging and diagnostic benchmark for advancing spatial reasoning research in VLMs. Our project page is at https://zesen01.github.io/jigsaw-puzzles.