Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning
This work addresses a critical gap in evaluating multi-modal AI models for researchers and developers, but it is incremental as it builds on existing benchmarking efforts by extending them to multi-image scenarios.
The authors tackled the lack of benchmarks for multi-image understanding in vision and language models by introducing the Multi-Image Relational Benchmark (MIRB), which evaluates models across perception, knowledge, reasoning, and multi-hop reasoning categories, and found that even state-of-the-art models like GPT-4V struggle significantly with these tasks, revealing a performance gap compared to single-image tasks.
The advancement of large language models (LLMs) has significantly broadened the scope of applications in natural language processing, with multi-modal LLMs extending these capabilities to integrate and interpret visual data. However, existing benchmarks for visual language models (VLMs) predominantly focus on single-image inputs, neglecting the crucial aspect of multi-image understanding. In this paper, we introduce a Multi-Image Relational Benchmark MIRB, designed to evaluate VLMs' ability to compare, analyze, and reason across multiple images. Our benchmark encompasses four categories: perception, visual world knowledge, reasoning, and multi-hop reasoning. Through a comprehensive evaluation of a wide range of open-source and closed-source models, we demonstrate that while open-source VLMs were shown to approach the performance of GPT-4V in single-image tasks, a significant performance gap remains in multi-image reasoning tasks. Our findings also reveal that even the state-of-the-art GPT-4V model struggles with our benchmark, underscoring the need for further research and development in this area. We believe our contribution of MIRB could serve as a testbed for developing the next-generation multi-modal models.