MapTab: Are MLLMs Ready for Multi-Criteria Route Planning in Heterogeneous Graphs?
This addresses the need for systematic evaluation of MLLMs in AGI development, though it is incremental as it focuses on benchmarking rather than proposing new methods.
The paper tackles the problem of evaluating Multimodal Large Language Models (MLLMs) for multi-criteria reasoning by introducing MapTab, a benchmark with 328 images and 196,800 route planning queries, and finds that current models face substantial challenges, with multimodal collaboration often underperforming unimodal approaches under limited visual perception.
Systematic evaluation of Multimodal Large Language Models (MLLMs) is crucial for advancing Artificial General Intelligence (AGI). However, existing benchmarks remain insufficient for rigorously assessing their reasoning capabilities under multi-criteria constraints. To bridge this gap, we introduce MapTab, a multimodal benchmark specifically designed to evaluate holistic multi-criteria reasoning in MLLMs via route planning tasks. MapTab requires MLLMs to perceive and ground visual cues from map images alongside route attributes (e.g., Time, Price) from structured tabular data. The benchmark encompasses two scenarios: Metromap, covering metro networks in 160 cities across 52 countries, and Travelmap, depicting 168 representative tourist attractions from 19 countries. In total, MapTab comprises 328 images, 196,800 route planning queries, and 3,936 QA queries, all incorporating 4 key criteria: Time, Price, Comfort, and Reliability. Extensive evaluations across 15 representative MLLMs reveal that current models face substantial challenges in multi-criteria multimodal reasoning. Notably, under conditions of limited visual perception, multimodal collaboration often underperforms compared to unimodal approaches. We believe MapTab provides a challenging and realistic testbed to advance the systematic evaluation of MLLMs. Our code is available at https://github.com/Ziqiao-Shang/MapTab.