An Agentic Approach Towards Replication Package Quality Evaluation
This work addresses the scalability bottleneck of manual artifact evaluation for software engineering researchers and reviewers, but the evaluation is preliminary (5 packages) and the approach is domain-specific.
The authors propose an agentic approach to automate replication package quality evaluation, translating 380 open-science guidelines into 51 criteria and operationalizing 31 for automated checks. Their multi-agent prototype achieves 91.4% inter-run consistency and 75.4% correctness on five packages, with a pilot survey indicating perceived usefulness.
Reproducibility in empirical software engineering relies on complete, accessible, and reusable research artifacts, yet artifact evaluation remains largely manual and difficult to scale. This emerging results paper explores an agentic approach for assessing replication package quality by translating open-science guidelines into machine-verifiable criteria. We consolidate 380 requirements from 34 sources into 51 reproducibility criteria, of which 31 are operationalized for automated artifact-based evaluation. Based on these criteria, we implement a multi-agent prototype that automatically inspects replication packages and produces evidence-grounded improvement reports. A preliminary evaluation on five replication packages shows high inter-run consistency of 91.4\% and 75.4\% correctness, through micro-averaged agreement with a manual baseline. The agent performs best on structural criteria such as code, environment, and artifact availability, but struggles with qualitative or mixed-method studies. A pilot survey with seven software engineering researchers indicates well perceived usefulness and adoption potential, while revealing cognitive load in the human-in-the-loop planning step. Overall, these emerging results indicate that agentic research artifact evaluation has the potential to support authors and reviewers by automating selected routine checks.