The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers
For the AI research community, this provides evidence that open science practices are improving, but the improvements predate formal checklists, suggesting a cultural shift rather than policy-driven change.
The study analyzed 56,800 papers from five leading AI conferences (2014-2024) and found that documentation practices improved significantly, with code and data sharing increasing from 11% to 64%, and estimated reproducibility rising from 28% to 64%.
The reproducibility crisis has directed the AI research community toward improving documentation practices. Several studies have identified methodological issues, and in response, the most impactful venues in the field have introduced reproducibility checklists. We seek to understand whether documentation practices have changed over time by assessing all published papers at five leading AI conferences over the past decade. Seven reproducibility variables were identified, quality-assured and used to analyse 56 800 publications. Our analysis reveals that in the period 2014 to 2024, documentation practices have improved; papers sharing both code and data increased nearly sixfold, from 11% to 64% Building on empirical reproducibility rates from a prior study, we estimate - inferred from documentation practices, not direct testing - that reproducibility increased from 28% in 2014 to 64% in 2024. Improvements in documentation practices predate the introduction of reproducibility checklists, suggesting these changes reflect a broader movement toward open science rather than a direct response to formal requirements.