SFBench: The SciFy Scientific Feasibility Benchmark
This benchmark addresses the need for evaluating AI systems on complex scientific reasoning tasks, specifically for assessing claim feasibility in materials science, a domain where existing benchmarks are limited.
SFBench is a benchmark dataset of 197 claims in materials science, each annotated with a feasibility score and explanation, designed to evaluate systems that assess scientific claim feasibility. Baseline results using GPT models show that current LLMs struggle with this task, achieving low agreement with expert annotations.
We present SFBench, a benchmark dataset for evaluating systems that assess the feasibility of scientific claims. SFBench includes 197 claims in materials science, each annotated with a ground-truth feasibility score on a five-point scale along with an explanation of that assessment. The collection differs from previous collections in several important ways: 1) it defines a complex task that requires reasoning over claims of varying scientific feasibility; 2) its claims are not extracted from existing scientific publications but are created de novo, greatly reducing the chances that LLMs have trained on them; 3) claims and ground truth are established by subject matter experts, not by artificial intelligence; and 4) unlike many benchmarks that ask about question/answer pairs, provide multiple choice answers, or ask questions requiring short, fixed answers, SFBench explanations are completely open-ended. We describe the benchmark design, data creation process, and evaluation metrics, and we report baseline results using recent GPT models.