Datasheets for AI and medical datasets (DAIMS): a data validation and documentation framework before machine learning analysis in medical research
This provides a standardized framework for researchers in medical ML to improve dataset preparation, though it is incremental as it builds on existing datasheet concepts.
The authors tackled the problem of inconsistent data validation and documentation in medical machine learning research by extending the 'Datasheets for Datasets' framework to create DAIMS, which includes a 24-item checklist, software tool, and flowchart for standardizing datasets and suggesting ML methods, resulting in a publicly available solution on GitHub and as an online app.
Despite progresses in data engineering, there are areas with limited consistencies across data validation and documentation procedures causing confusions and technical problems in research involving machine learning. There have been progresses by introducing frameworks like "Datasheets for Datasets", however there are areas for improvements to prepare datasets, ready for ML pipelines. Here, we extend the framework to "Datasheets for AI and medical datasets - DAIMS." Our publicly available solution, DAIMS, provides a checklist including data standardization requirements, a software tool to assist the process of the data preparation, an extended form for data documentation and pose research questions, a table as data dictionary, and a flowchart to suggest ML analyses to address the research questions. The checklist consists of 24 common data standardization requirements, where the tool checks and validate a subset of them. In addition, we provided a flowchart mapping research questions to suggested ML methods. DAIMS can serve as a reference for standardizing datasets and a roadmap for researchers aiming to apply effective ML techniques in their medical research endeavors. DAIMS is available on GitHub and as an online app to automate key aspects of dataset evaluation, facilitating efficient preparation of datasets for ML studies.