AstroMMBench: A Benchmark for Evaluating Multimodal Large Language Models Capabilities in Astronomy
This addresses the problem of lacking domain-specific benchmarks for MLLMs in astronomy, which is incremental as it extends existing benchmarking efforts to a specialized scientific field.
The authors tackled the challenge of evaluating multimodal large language models (MLLMs) on astronomical image interpretation by introducing AstroMMBench, a comprehensive benchmark with 621 multiple-choice questions across six astrophysical subfields, and found that Ovis2-34B achieved the highest overall accuracy of 70.5% among 25 tested models.
Astronomical image interpretation presents a significant challenge for applying multimodal large language models (MLLMs) to specialized scientific tasks. Existing benchmarks focus on general multimodal capabilities but fail to capture the complexity of astronomical data. To bridge this gap, we introduce AstroMMBench, the first comprehensive benchmark designed to evaluate MLLMs in astronomical image understanding. AstroMMBench comprises 621 multiple-choice questions across six astrophysical subfields, curated and reviewed by 15 domain experts for quality and relevance. We conducted an extensive evaluation of 25 diverse MLLMs, including 22 open-source and 3 closed-source models, using AstroMMBench. The results show that Ovis2-34B achieved the highest overall accuracy (70.5%), demonstrating leading capabilities even compared to strong closed-source models. Performance showed variations across the six astrophysical subfields, proving particularly challenging in domains like cosmology and high-energy astrophysics, while models performed relatively better in others, such as instrumentation and solar astrophysics. These findings underscore the vital role of domain-specific benchmarks like AstroMMBench in critically evaluating MLLM performance and guiding their targeted development for scientific applications. AstroMMBench provides a foundational resource and a dynamic tool to catalyze advancements at the intersection of AI and astronomy.