Does the Market of Citations Reward Reproducible Work?
This addresses the incentive structure in academia for researchers, showing that reproducibility may be rewarded in some fields but not universally, which is incremental as it builds on prior limited evidence.
The study investigated whether reproducible academic work receives more citations, finding that in fields like Medicine and Machine Learning, reproducible work correlates with higher citations, while other fields show no relationship, and that code availability and thorough referencing also positively correlate with citations.
The field of bibliometrics, studying citations and behavior, is critical to the discussion of reproducibility. Citations are one of the primary incentive and reward systems for academic work, and so we desire to know if this incentive rewards reproducible work. Yet to the best of our knowledge, only one work has attempted to look at this combined space, concluding that non-reproducible work is more highly cited. We show that answering this question is more challenging than first proposed, and subtle issues can inhibit a robust conclusion. To make inferences with more robust behavior, we propose a hierarchical Bayesian model that incorporates the citation rate over time, rather than the total number of citations after a fixed amount of time. In doing so we show that, under current evidence the answer is more likely that certain fields of study such as Medicine and Machine Learning (ML) do correlate reproducible works with more citations, but other fields appear to have no relationship. Further, we find that making code available and thoroughly referencing prior works appear to also positively correlate with increased citations. Our code and data can be found at https://github.com/EdwardRaff/ReproducibleCitations .