Was that so hard? Estimating human classification difficulty
This work addresses the need for ordering training cases by difficulty to improve doctor training efficiency in medical diagnosis, though it is incremental as it builds on existing metric learning techniques.
The paper tackled the problem of estimating the difficulty of medical image classification for doctors, introducing methods based on deep metric learning that work with or without ground truth difficulty labels, and achieved high Kendall rank correlation coefficients on two medical datasets, outperforming existing methods by a large margin.
When doctors are trained to diagnose a specific disease, they learn faster when presented with cases in order of increasing difficulty. This creates the need for automatically estimating how difficult it is for doctors to classify a given case. In this paper, we introduce methods for estimating how hard it is for a doctor to diagnose a case represented by a medical image, both when ground truth difficulties are available for training, and when they are not. Our methods are based on embeddings obtained with deep metric learning. Additionally, we introduce a practical method for obtaining ground truth human difficulty for each image case in a dataset using self-assessed certainty. We apply our methods to two different medical datasets, achieving high Kendall rank correlation coefficients, showing that we outperform existing methods by a large margin on our problem and data.