MapIQ: Evaluating Multimodal Large Language Models for Map Question Answering
This work addresses the problem of evaluating MLLMs on diverse map types for researchers in AI and cartography, but it is incremental as it extends existing Map-VQA benchmarks.
The authors tackled the limited scope of map visual question answering (Map-VQA) by introducing MapIQ, a benchmark dataset with 14,706 question-answer pairs across three map types and six themes, and evaluated multiple multimodal large language models (MLLMs) against a human baseline, finding insights into their robustness and sensitivity to design changes.
Recent advancements in multimodal large language models (MLLMs) have driven researchers to explore how well these models read data visualizations, e.g., bar charts, scatter plots. More recently, attention has shifted to visual question answering with maps (Map-VQA). However, Map-VQA research has primarily focused on choropleth maps, which cover only a limited range of thematic categories and visual analytical tasks. To address these gaps, we introduce MapIQ, a benchmark dataset comprising 14,706 question-answer pairs across three map types: choropleth maps, cartograms, and proportional symbol maps spanning topics from six distinct themes (e.g., housing, crime). We evaluate multiple MLLMs using six visual analytical tasks, comparing their performance against one another and a human baseline. An additional experiment examining the impact of map design changes (e.g., altered color schemes, modified legend designs, and removal of map elements) provides insights into the robustness and sensitivity of MLLMs, their reliance on internal geographic knowledge, and potential avenues for improving Map-VQA performance.