A Siren Song of Open Source Reproducibility
This addresses the problem of ineffective reproducibility practices for the machine learning research community, highlighting an incremental critique of current conference strategies.
The paper argues that requiring code submission for reproducibility in conferences is misguided and potentially harmful, advocating for more evidence-based actions to advance reproducible machine learning research.
As reproducibility becomes a greater concern, conferences have largely converged to a strategy of asking reviewers to indicate whether code was attached to a submission. This is part of a larger trend of taking action based on assumed ideals, without studying if those actions will yield the desired outcome. Our argument is that this focus on code for replication is misguided if we want to improve the state of reproducible research. This focus can be harmful -- we should not force code to be submitted. There is a lack of evidence for effective actions taken by conferences to encourage and reward reproducibility. We argue that venues must take more action to advance reproducible machine learning research today.