VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning
This addresses the problem of assessing MLLMs' high-level reasoning abilities for researchers and developers, but it is incremental as it focuses on evaluation rather than model improvement.
The paper tackles the challenge of evaluating Multimodal Large Language Models (MLLMs) for abstract reasoning across images by introducing VOILA, a benchmark for perceptual understanding and analogical reasoning, showing that current models struggle with accuracy as low as 13% for challenging tasks and 29% for simpler ones, compared to 70% human performance.
Multimodal Large Language Models (MLLMs) have become a powerful tool for integrating visual and textual information. Despite their exceptional performance on visual understanding benchmarks, measuring their ability to reason abstractly across multiple images remains a significant challenge. To address this, we introduce VOILA, a large-scale, open-ended, dynamic benchmark designed to evaluate MLLMs' perceptual understanding and abstract relational reasoning. VOILA employs an analogical mapping approach in the visual domain, requiring models to generate an image that completes an analogy between two given image pairs, reference and application, without relying on predefined choices. Our experiments demonstrate that the analogical reasoning tasks in VOILA present a challenge to MLLMs. Through multi-step analysis, we reveal that current MLLMs struggle to comprehend inter-image relationships and exhibit limited capabilities in high-level relational reasoning. Notably, we observe that performance improves when following a multi-step strategy of least-to-most prompting. Comprehensive evaluations on open-source models and GPT-4o show that on text-based answers, the best accuracy for challenging scenarios is 13% (LLaMa 3.2) and even for simpler tasks is only 29% (GPT-4o), while human performance is significantly higher at 70% across both difficulty levels.