Beyond Perception: Evaluating Abstract Visual Reasoning through Multi-Stage Task
This work addresses a gap in evaluating higher-order reasoning for AI researchers, though it is incremental as it builds on existing benchmarks like RAVEN.
The paper tackles the problem of evaluating Abstract Visual Reasoning (AVR) in Multimodal Large Language Models (MLLMs) by introducing MultiStAR, a multi-stage benchmark based on RAVEN, and MSEval, a metric that assesses intermediate steps. Results show that MLLMs perform well on basic perception but struggle with complex rule detection stages, with specific performance gaps highlighted across 17 models.
Current Multimodal Large Language Models (MLLMs) excel in general visual reasoning but remain underexplored in Abstract Visual Reasoning (AVR), which demands higher-order reasoning to identify abstract rules beyond simple perception. Existing AVR benchmarks focus on single-step reasoning, emphasizing the end result but neglecting the multi-stage nature of reasoning process. Past studies found MLLMs struggle with these benchmarks, but it doesn't explain how they fail. To address this gap, we introduce MultiStAR, a Multi-Stage AVR benchmark, based on RAVEN, designed to assess reasoning across varying levels of complexity. Additionally, existing metrics like accuracy only focus on the final outcomes while do not account for the correctness of intermediate steps. Therefore, we propose a novel metric, MSEval, which considers the correctness of intermediate steps in addition to the final outcomes. We conduct comprehensive experiments on MultiStAR using 17 representative close-source and open-source MLLMs. The results reveal that while existing MLLMs perform adequately on basic perception tasks, they continue to face challenges in more complex rule detection stages.