An End-to-End System for Reproducibility Assessment of Source Code Repositories via Their Readmes
This work addresses the need for automated reproducibility assessment in ML research, though it is incremental as it builds on existing templates and methods.
The authors tackled the problem of assessing reproducibility of machine learning research by developing an end-to-end system that evaluates source code repository Readme files against a template, with their section similarity-based method outperforming a hierarchical transformer model and offering better explainability.
Increased reproducibility of machine learning research has been a driving force for dramatic improvements in learning performances. The scientific community further fosters this effort by including reproducibility ratings in reviewer forms and considering them as a crucial factor for the overall evaluation of papers. Accompanying source code is not sufficient to make a work reproducible. The shared codes should meet the ML reproducibility checklist as well. This work aims to support reproducibility evaluations of papers with source codes. We propose an end-to-end system that operates on the Readme file of the source code repositories. The system checks the compliance of a given Readme to a template proposed by a widely used platform for sharing source codes of research. Our system generates scores based on a custom function to combine section scores. We also train a hierarchical transformer model to assign a class label to a given Readme. The experimental results show that the section similarity-based system performs better than the hierarchical transformer. Moreover, it has an advantage regarding explainability since one can directly relate the score to the sections of Readme files.