Metric Hub: A metric library and practical selection workflow for use-case-driven data quality assessment in medical AI
This work addresses the need for practical data quality assessment to support regulatory approval and adoption of AI in medicine, but it is incremental as it builds on an existing theoretical framework.
The authors tackled the problem of quantifying data quality for trustworthy medical AI by introducing Metric Hub, a library and workflow for selecting data quality metrics based on use cases, and demonstrated its application on the PTB-XL ECG dataset.
Machine learning (ML) in medicine has transitioned from research to concrete applications aimed at supporting several medical purposes like therapy selection, monitoring and treatment. Acceptance and effective adoption by clinicians and patients, as well as regulatory approval, require evidence of trustworthiness. A major factor for the development of trustworthy AI is the quantification of data quality for AI model training and testing. We have recently proposed the METRIC-framework for systematically evaluating the suitability (fit-for-purpose) of data for medical ML for a given task. Here, we operationalize this theoretical framework by introducing a collection of data quality metrics - the metric library - for practically measuring data quality dimensions. For each metric, we provide a metric card with the most important information, including definition, applicability, examples, pitfalls and recommendations, to support the understanding and implementation of these metrics. Furthermore, we discuss strategies and provide decision trees for choosing an appropriate set of data quality metrics from the metric library given specific use cases. We demonstrate the impact of our approach exemplarily on the PTB-XL ECG-dataset. This is a first step to enable fit-for-purpose evaluation of training and test data in practice as the base for establishing trustworthy AI in medicine.