MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understanding
This addresses the need for evolving benchmarks in scientific AI evaluation, though it is incremental as it builds on existing MLLM capabilities.
The authors tackled the problem of evaluating high-level scientific understanding in multimodal large language models (MLLMs) by introducing MAC, a live benchmark with over 25,000 image-text pairs from top journals, and found that MLLMs have strong perceptual abilities but limited cross-modal scientific reasoning, with their proposed DAD method improving performance by up to 11%.
As multimodal large language models (MLLMs) grow increasingly capable, fixed benchmarks are gradually losing their effectiveness in evaluating high-level scientific understanding. In this paper, we introduce the Multimodal Academic Cover benchmark (MAC), a live benchmark that could continuously evolve with scientific advancement and model progress. MAC leverages over 25,000 image-text pairs sourced from issues of top-tier scientific journals such as Nature, Science, and Cell, challenging MLLMs to reason across abstract visual and textual scientific content. Experiments on our most recent yearly snapshot, MAC-2025, reveal that while MLLMs demonstrate strong perceptual abilities, their cross-modal scientific reasoning remains limited. To bridge this gap, we propose DAD, a lightweight inference-time approach that enhances MLLMs by extending MLLM visual features with language space reasoning, achieving performance improvements of up to 11%. Finally, we highlight the live nature of MAC through experiments on updating journal covers and models for curation, illustrating its potential to remain aligned with the frontier of human knowledge. We release our benchmark at https://github.com/mhjiang0408/MAC_Bench.