MatViX: Multimodal Information Extraction from Visually Rich Articles
This addresses the challenge of accelerating materials discovery by enabling better information extraction from complex scientific documents, though it is incremental as it focuses on benchmarking and evaluation.
The paper tackles the problem of extracting structured information from multimodal scientific articles in materials science, introducing the MatViX benchmark with 324 articles and 1,688 JSON files, and shows that current vision-language models have significant room for improvement in accuracy.
Multimodal information extraction (MIE) is crucial for scientific literature, where valuable data is often spread across text, figures, and tables. In materials science, extracting structured information from research articles can accelerate the discovery of new materials. However, the multimodal nature and complex interconnections of scientific content present challenges for traditional text-based methods. We introduce \textsc{MatViX}, a benchmark consisting of $324$ full-length research articles and $1,688$ complex structured JSON files, carefully curated by domain experts. These JSON files are extracted from text, tables, and figures in full-length documents, providing a comprehensive challenge for MIE. We introduce an evaluation method to assess the accuracy of curve similarity and the alignment of hierarchical structures. Additionally, we benchmark vision-language models (VLMs) in a zero-shot manner, capable of processing long contexts and multimodal inputs, and show that using a specialized model (DePlot) can improve performance in extracting curves. Our results demonstrate significant room for improvement in current models. Our dataset and evaluation code are available\footnote{\url{https://matvix-bench.github.io/}}.