Investigating the Quality of DermaMNIST and Fitzpatrick17k Dermatological Image Datasets
This work addresses data quality problems in dermatology datasets, which is crucial for developing accurate deep learning models in medical diagnostics, though it is incremental as it focuses on identifying and fixing existing issues.
The paper analyzed data quality issues like duplicates and mislabeling in dermatological image datasets DermaMNIST, HAM10000, and Fitzpatrick17k, and measured their impact on benchmark results, proposing corrections to improve reliability.
The remarkable progress of deep learning in dermatological tasks has brought us closer to achieving diagnostic accuracies comparable to those of human experts. However, while large datasets play a crucial role in the development of reliable deep neural network models, the quality of data therein and their correct usage are of paramount importance. Several factors can impact data quality, such as the presence of duplicates, data leakage across train-test partitions, mislabeled images, and the absence of a well-defined test partition. In this paper, we conduct meticulous analyses of three popular dermatological image datasets: DermaMNIST, its source HAM10000, and Fitzpatrick17k, uncovering these data quality issues, measure the effects of these problems on the benchmark results, and propose corrections to the datasets. Besides ensuring the reproducibility of our analysis, by making our analysis pipeline and the accompanying code publicly available, we aim to encourage similar explorations and to facilitate the identification and addressing of potential data quality issues in other large datasets.