VGRP-Bench: Visual Grid Reasoning Puzzle Benchmark for Large Vision-Language Models
This work addresses the need for better evaluation and enhancement of structured reasoning in LVLMs, which is crucial for real-world problem-solving, though it is incremental as it builds on existing benchmark and fine-tuning approaches.
The authors tackled the problem of large vision-language models (LVLMs) struggling with visual grid reasoning puzzles by introducing VGRP-Bench, a benchmark with 20 diverse puzzles, and found that even state-of-the-art models like GPT-4o and Gemini-Thinking perform poorly, with supervised fine-tuning strategies improving performance on trained puzzles but showing limited generalization to unseen ones.
Large Vision-Language Models (LVLMs) struggle with puzzles, which require precise perception, rule comprehension, and logical reasoning. Assessing and enhancing their performance in this domain is crucial, as it reflects their ability to engage in structured reasoning - an essential skill for real-world problem-solving. However, existing benchmarks primarily evaluate pre-trained models without additional training or fine-tuning, often lack a dedicated focus on reasoning, and fail to establish a systematic evaluation framework. To address these limitations, we introduce VGRP-Bench, a Visual Grid Reasoning Puzzle Benchmark featuring 20 diverse puzzles. VGRP-Bench spans multiple difficulty levels, and includes extensive experiments not only on existing chat LVLMs (e.g., GPT-4o), but also on reasoning LVLMs (e.g., Gemini-Thinking). Our results reveal that even the state-of-the-art LVLMs struggle with these puzzles, highlighting fundamental limitations in their puzzle-solving capabilities. Most importantly, through systematic experiments, we identify and analyze key factors influencing LVLMs' puzzle-solving performance, including the number of clues, grid size, and rule complexity. Furthermore, we explore two Supervised Fine-Tuning (SFT) strategies that can be used in post-training: SFT on solutions (S-SFT) and SFT on synthetic reasoning processes (R-SFT). While both methods significantly improve performance on trained puzzles, they exhibit limited generalization to unseen ones. We will release VGRP-Bench to facilitate further research on LVLMs for complex, real-world problem-solving. Project page: https://yufan-ren.com/subpage/VGRP-Bench/.