Oedipus and the Sphinx: Benchmarking and Improving Visual Language Models for Complex Graphic Reasoning
This work addresses the lack of benchmarks for complex graphic reasoning in VLMs, which is an incremental advancement for researchers in AI and computer vision.
The authors tackled the problem of evaluating visual language models (VLMs) in complex graphic reasoning by proposing ReasonBench, a benchmark with 1,613 questions from real-world intelligence tests, and revealed significant limitations in current models. They introduced a dual optimization strategy that improved VLM performance by 33.5%.
Evaluating the performance of visual language models (VLMs) in graphic reasoning tasks has become an important research topic. However, VLMs still show obvious deficiencies in simulating human-level graphic reasoning capabilities, especially in complex graphic reasoning and abstract problem solving, which are less studied and existing studies only focus on simple graphics. To evaluate the performance of VLMs in complex graphic reasoning, we propose ReasonBench, the first evaluation benchmark focused on structured graphic reasoning tasks, which includes 1,613 questions from real-world intelligence tests. ReasonBench covers reasoning dimensions related to location, attribute, quantity, and multi-element tasks, providing a comprehensive evaluation of the performance of VLMs in spatial, relational, and abstract reasoning capabilities. We benchmark 11 mainstream VLMs (including closed-source and open-source models) and reveal significant limitations of current models. Based on these findings, we propose a dual optimization strategy: Diagrammatic Reasoning Chain (DiaCoT) enhances the interpretability of reasoning by decomposing layers, and ReasonTune enhances the task adaptability of model reasoning through training, all of which improves VLM performance by 33.5\%. All experimental data and code are in the repository: https://huggingface.co/datasets/cistine/ReasonBench.