Let's agree to disagree: learning highly debatable multirater labelling
This addresses the challenge of inconsistent labeling in medical imaging for radiologists, though it is incremental as it builds on existing consensus modeling approaches.
The paper tackled the problem of high variability in labeling small pathological objects by human raters in radiology, by jointly modeling individual rater behavior and multirater consensus in deep learning. Results showed significant performance improvements compared to directly predicting consensus labels, while enabling characterization of rater consistency.
Classification and differentiation of small pathological objects may greatly vary among human raters due to differences in training, expertise and their consistency over time. In a radiological setting, objects commonly have high within-class appearance variability whilst sharing certain characteristics across different classes, making their distinction even more difficult. As an example, markers of cerebral small vessel disease, such as enlarged perivascular spaces (EPVS) and lacunes, can be very varied in their appearance while exhibiting high inter-class similarity, making this task highly challenging for human raters. In this work, we investigate joint models of individual rater behaviour and multirater consensus in a deep learning setting, and apply it to a brain lesion object-detection task. Results show that jointly modelling both individual and consensus estimates leads to significant improvements in performance when compared to directly predicting consensus labels, while also allowing the characterization of human-rater consistency.