SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verification
This addresses the problem of evaluating multimodal AI for scientific verification, providing a new benchmark for researchers, but it is incremental as it builds on existing verification tasks.
The authors introduced SciVer, a benchmark with 3,000 expert-annotated examples from 1,113 scientific papers to evaluate multimodal foundation models on scientific claim verification, finding a substantial performance gap between 21 state-of-the-art models and human experts.
We introduce SciVer, the first benchmark specifically designed to evaluate the ability of foundation models to verify claims within a multimodal scientific context. SciVer consists of 3,000 expert-annotated examples over 1,113 scientific papers, covering four subsets, each representing a common reasoning type in multimodal scientific claim verification. To enable fine-grained evaluation, each example includes expert-annotated supporting evidence. We assess the performance of 21 state-of-the-art multimodal foundation models, including o4-mini, Gemini-2.5-Flash, Llama-3.2-Vision, and Qwen2.5-VL. Our experiment reveals a substantial performance gap between these models and human experts on SciVer. Through an in-depth analysis of retrieval-augmented generation (RAG), and human-conducted error evaluations, we identify critical limitations in current open-source models, offering key insights to advance models' comprehension and reasoning in multimodal scientific literature tasks.