MANBench: Is Your Multimodal Model Smarter than Human?
This work addresses the need for rigorous evaluation of MLLMs compared to human abilities, providing a benchmark to inspire improvements, though it is incremental as it builds on existing benchmarking efforts.
The authors tackled the problem of evaluating whether multimodal large language models (MLLMs) can surpass human performance by introducing MANBench, a bilingual benchmark with 1,314 questions across nine tasks, and found that while MLLMs excel in some areas like Knowledge and Text-Image Understanding, they struggle with deeper cross-modal reasoning tasks and fall short of human-level performance in many domains.
The rapid advancement of Multimodal Large Language Models (MLLMs) has ignited discussions regarding their potential to surpass human performance in multimodal tasks. In response, we introduce MANBench (Multimodal Ability Norms Benchmark), a bilingual benchmark (English and Chinese) comprising 1,314 questions across nine tasks, spanning knowledge-based and non-knowledge-based domains. MANBench emphasizes intuitive reasoning, seamless cross-modal integration, and real-world complexity, providing a rigorous evaluation framework. Through extensive human experiments involving diverse participants, we compared human performance against state-of-the-art MLLMs. The results indicate that while MLLMs excel in tasks like Knowledge and Text-Image Understanding, they struggle with deeper cross-modal reasoning tasks such as Transmorphic Understanding, Image Consistency, and Multi-image Understanding. Moreover, both humans and MLLMs face challenges in highly complex tasks like Puzzles and Spatial Imagination. MANBench highlights the strengths and limitations of MLLMs, revealing that even advanced models fall short of achieving human-level performance across many domains. We hope MANBench will inspire efforts to bridge the gap between MLLMs and human multimodal capabilities. The code and dataset are available at https://github.com/micdz/MANBench.