SciCoQA: Quality Assurance for Scientific Paper--Code Alignment
This addresses the problem of ensuring faithful implementations for researchers and practitioners in computational science, but it is incremental as it builds on existing datasets and methods.
The authors tackled the problem of detecting discrepancies between scientific papers and their codebases by introducing SciCoQA, a dataset of 611 paper-code discrepancies (81 real, 530 synthetic) across disciplines like AI and Physics. They found that even the best model, GPT-5, could only detect 45.7% of real-world discrepancies, highlighting the task's difficulty.
We present SciCoQA, a dataset for detecting discrepancies between scientific publications and their codebases to ensure faithful implementations. We construct SciCoQA from GitHub issues and reproducibility papers, and to scale our dataset, we propose a synthetic data generation method for constructing paper-code discrepancies. We analyze the paper-code discrepancies in detail and propose discrepancy types and categories to better understand the occurring mismatches. In total, our dataset consists of 611 paper-code discrepancies (81 real, 530 synthetic), spanning diverse computational science disciplines, including AI, Physics, Quantitative Biology, and others. Our evaluation of 21 LLMs highlights the difficulty of SciCoQA, particularly for instances involving omitted paper details, long-context inputs, and data outside the models' pre-training corpus. The best performing model in our evaluation, GPT-5, can only detect 45.7\% of real-world paper-code discrepancies.