ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review
For peer reviewers and the scientific community, ARA provides a scalable method to assess reproducibility, addressing a key bottleneck in modern peer review.
ARA formalizes reproducibility assessment as a structured reasoning task over scientific documents, extracting workflow graphs and evaluating reconstructability. It achieves ~61% accuracy on three benchmarks, outperforming prior methods on ReproBench (60.71% vs. 36.84%) and GoldStandardDB (61.68% vs. 43.56%).
Scientific peer review increasingly struggles to assess reproducibility at the scale and complexity of modern research output. Evaluating reproducibility requires reconstructing experimental dependencies, methodological choices, data flows, and result-generating procedures, which often exceeds what human reviewers can provide. Agentic Reproducibility Assessment (ARA) formalizes reproducibility assessment as a structured reasoning task over scientific documents. Given a paper, ARA extracts a directed workflow graph linking sources, methods, experiments, and outputs, then evaluates its reconstructability using structural and content-based scores for reproducibility assessments. Experiments on 213 ReScience C articles - the largest cross-domain benchmark of human-validated computational reproducibility studies considered to date - demonstrate ARA's generalizability and consistent workflow reconstruction and assessment across LLMs, model temperatures, and scientific domains. ARA achieves ~61% accuracy on three benchmarks, and the highest accuracy reported on ReproBench (60.71% vs. 36.84%) and GoldStandardDB (61.68% vs. 43.56%), highlighting its potential to complement human review at scale and enabling next-generation peer review. Code and Data available: https://github.com/AndresLaverdeMarin/agentic_reproducibility_assessment.