SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables
This addresses the need for better benchmarks in scientific claim verification, though it is incremental as it builds on existing fact-checking efforts.
The authors tackled the problem of scientific fact-checking by introducing SCITAB, a dataset of 1.2K expert-verified claims from scientific publications that require compositional reasoning, and found that state-of-the-art models, except GPT-4, performed barely above random guessing.
Current scientific fact-checking benchmarks exhibit several shortcomings, such as biases arising from crowd-sourced claims and an over-reliance on text-based evidence. We present SCITAB, a challenging evaluation dataset consisting of 1.2K expert-verified scientific claims that 1) originate from authentic scientific publications and 2) require compositional reasoning for verification. The claims are paired with evidence-containing scientific tables annotated with labels. Through extensive evaluations, we demonstrate that SCITAB poses a significant challenge to state-of-the-art models, including table-based pretraining models and large language models. All models except GPT-4 achieved performance barely above random guessing. Popular prompting techniques, such as Chain-of-Thought, do not achieve much performance gains on SCITAB. Our analysis uncovers several unique challenges posed by SCITAB, including table grounding, claim ambiguity, and compositional reasoning. Our codes and data are publicly available at https://github.com/XinyuanLu00/SciTab.