On the Readiness of Scientific Data for a Fair and Transparent Use in Machine Learning
This work addresses the problem of ensuring fair and transparent ML systems by evaluating and improving data documentation practices, which is incremental as it builds on existing legislative and research efforts.
The study analyzed 4041 data papers across domains to assess how well scientific data documentation meets ML fairness and transparency needs, finding gaps in completeness and coverage and proposing recommendation guidelines for data creators and publishers.
To ensure the fairness and trustworthiness of machine learning (ML) systems, recent legislative initiatives and relevant research in the ML community have pointed out the need to document the data used to train ML models. Besides, data-sharing practices in many scientific domains have evolved in recent years for reproducibility purposes. In this sense, academic institutions' adoption of these practices has encouraged researchers to publish their data and technical documentation in peer-reviewed publications such as data papers. In this study, we analyze how this broader scientific data documentation meets the needs of the ML community and regulatory bodies for its use in ML technologies. We examine a sample of 4041 data papers of different domains, assessing their completeness, coverage of the requested dimensions, and trends in recent years. We focus on the most and least documented dimensions and compare the results with those of an ML-focused venue (NeurIPS D&B track) publishing papers describing datasets. As a result, we propose a set of recommendation guidelines for data creators and scientific data publishers to increase their data's preparedness for its transparent and fairer use in ML technologies.