Abstract
The reproducibility crisis has directed the AI research community toward improving documentation practices. Several studies have identified methodological issues, and in response, the most impactful venues in the field have introduced reproducibility checklists. We seek to understand whether documentation practices have changed over time by assessing all published papers at five leading AI conferences over the past decade. Seven reproducibility variables were identified, quality-assured, and used to analyze 56,800 publications.
Our analysis reveals that from 2014 to 2024, documentation practices have improved; papers sharing both code and data increased nearly sixfold, from 11% to 64%. Building on empirical reproducibility rates from a prior study, we estimate—based on documentation practices, not direct testing—that reproducibility increased from 28% in 2014 to 64% in 2024. Improvements in documentation practices predate the introduction of reproducibility checklists, suggesting these changes reflect a broader movement toward open science rather than a direct response to formal requirements.
Blogger's Review: This article quantifies the improvements in documentation practices, showcasing the AI research community's commitment to reproducibility issues. The findings underscore the importance of transparency and sharing, providing vital references for future research while reflecting the gradual implementation of open science principles.