Benchmarking Cross-Scale Perception Ability of Large Multimodal Models in Material Science
This addresses a gap in benchmarking for materials science researchers, though it is incremental as it focuses on dataset creation and evaluation rather than novel model development.
The authors tackled the lack of benchmarks for evaluating Large Multimodal Models' ability to reason across physical scales in materials science by introducing CSMBench, a dataset of 1,041 figures from premier journals, and found that model performance varies significantly across atomic, micro, meso, and macro scales.
Unraveling the hierarchical structure-property relationships is the central challenge of materials science, necessitating the interpretation of data across vast physical scales from micro to macro. Despite the rapid integration of Large Multimodal Models (LMMs) into scientific workflows, existing scientific benchmarks primarily focus on general chart interpretation or isolated common-sense reasoning, failing to capture reasoning ability across intricate physical dimensions. To address this, we introduce CSMBench, a dataset comprising 1,041 high-quality figures curated from premier journals up to September 2025. CSMBench categorizes data into four scientifically distinct regimes: atomic, micro, meso, and macro scales, strictly aligning with the focus and definitions in materials study. Through open-ended figure description and multiple-choice caption matching tasks, we evaluate state-of-the-art open-source and closed-source models. Our analysis identifies that performance varies significantly across physical scales due to the distinct visual characteristics, highlighting the limitations of current generalist models and identifying critical directions for achieving hierarchical and accurate understanding in materials research. The CSMBench is publicly released at: https://huggingface.co/datasets/lututu/CSMBench.