Can Multimodal LLMs See Materials Clearly? A Multimodal Benchmark on Materials Characterization
This work addresses the limited adaptability of MLLMs for materials scientists in real-world characterization scenarios, though it is incremental as it primarily introduces a benchmark rather than a new method.
The authors tackled the problem of evaluating multimodal large language models (MLLMs) on materials characterization imaging data by introducing MatCha, a benchmark with 1,500 expert-level questions, and found that state-of-the-art MLLMs show a significant performance gap compared to human experts, with degradation in tasks requiring higher-level expertise.
Materials characterization is fundamental to acquiring materials information, revealing the processing-microstructure-property relationships that guide material design and optimization. While multimodal large language models (MLLMs) have recently shown promise in generative and predictive tasks within materials science, their capacity to understand real-world characterization imaging data remains underexplored. To bridge this gap, we present MatCha, the first benchmark for materials characterization image understanding, comprising 1,500 questions that demand expert-level domain expertise. MatCha encompasses four key stages of materials research comprising 21 distinct tasks, each designed to reflect authentic challenges faced by materials scientists. Our evaluation of state-of-the-art MLLMs on MatCha reveals a significant performance gap compared to human experts. These models exhibit degradation when addressing questions requiring higher-level expertise and sophisticated visual perception. Simple few-shot and chain-of-thought prompting struggle to alleviate these limitations. These findings highlight that existing MLLMs still exhibit limited adaptability to real-world materials characterization scenarios. We hope MatCha will facilitate future research in areas such as new material discovery and autonomous scientific agents. MatCha is available at https://github.com/FreedomIntelligence/MatCha.