ExpVid: A Benchmark for Experiment Video Understanding & Reasoning
This provides a diagnostic tool for developing MLLMs to assist in scientific experimentation, though it is incremental as it focuses on benchmarking rather than a new method.
The authors tackled the problem of evaluating Multimodal Large Language Models (MLLMs) on scientific experiment videos by introducing ExpVid, a benchmark with a three-level task hierarchy, and found that while MLLMs excel at coarse-grained recognition, they struggle with fine details, state tracking, and linking procedures to outcomes, revealing a performance gap between proprietary and open-source models.
Multimodal Large Language Models (MLLMs) hold promise for accelerating scientific discovery by interpreting complex experimental procedures. However, their true capabilities are poorly understood, as existing benchmarks neglect the fine-grained and long-horizon nature of authentic laboratory work, especially in wet-lab settings. To bridge this gap, we introduce ExpVid, the first benchmark designed to systematically evaluate MLLMs on scientific experiment videos. Curated from peer-reviewed video publications, ExpVid features a new three-level task hierarchy that mirrors the scientific process: (1) Fine-grained Perception of tools, materials, and actions; (2) Procedural Understanding of step order and completeness; and (3) Scientific Reasoning that connects the full experiment to its published conclusions. Our vision-centric annotation pipeline, combining automated generation with multi-disciplinary expert validation, ensures that tasks require visual grounding. We evaluate 19 leading MLLMs on ExpVid and find that while they excel at coarse-grained recognition, they struggle with disambiguating fine details, tracking state changes over time, and linking experimental procedures to scientific outcomes. Our results reveal a notable performance gap between proprietary and open-source models, particularly in high-order reasoning. ExpVid not only provides a diagnostic tool but also charts a roadmap for developing MLLMs capable of becoming trustworthy partners in scientific experimentation.