Crowd disagreement about medical images is informative
This work addresses the problem of potentially losing useful information in medical image analysis by ignoring annotator disagreement, though it is incremental as it builds on existing consensus-labeling methods.
The study investigated whether disagreement among crowd annotators in medical image classification is informative, using skin lesion data from the ISIC 2017 dataset to predict melanoma. Results showed that while mean annotations performed best, disagreement measures still provided valuable information.
Classifiers for medical image analysis are often trained with a single consensus label, based on combining labels given by experts or crowds. However, disagreement between annotators may be informative, and thus removing it may not be the best strategy. As a proof of concept, we predict whether a skin lesion from the ISIC 2017 dataset is a melanoma or not, based on crowd annotations of visual characteristics of that lesion. We compare using the mean annotations, illustrating consensus, to standard deviations and other distribution moments, illustrating disagreement. We show that the mean annotations perform best, but that the disagreement measures are still informative. We also make the crowd annotations used in this paper available at \url{https://figshare.com/s/5cbbce14647b66286544}.