Research Artifacts in Secondary Studies: A Systematic Mapping in Software Engineering
For researchers conducting secondary studies in software engineering, this work highlights the current low availability of research artifacts and advocates for mandatory publication to improve transparency and reproducibility.
This paper evaluates how systematic reviews in software engineering report research artifacts, finding that only 31.5% of 537 studies include them, with a gradual improvement over time but still only 62.0% in 2023, and just 30.4% using a permanent repository with a DOI.
Context: Systematic reviews (SRs) summarize state-of-the-art evidence in science, including software engineering (SE). Objective: Our objective is to evaluate how SRs report research artifacts and to provide a comprehensive list of these artifacts. Method: We examined 537 secondary studies published between 2013 and 2023 to analyze the availability and reporting of research artifacts. Results: Our findings indicate that only 31.5% of the reviewed studies include research artifacts. Encouragingly, the situation is gradually improving, as our regression analysis shows a significant increase in the availability of research artifacts over time. However, in 2023, just 62.0% of secondary studies provide a research artifact while an even lower percentage, 30.4% use a permanent repository with a digital object identifier (DOI) for storage. Conclusion: To enhance transparency and reproducibility in SE research, we advocate for the mandatory publication of research artifacts in secondary studies.