Modeling Multiple Annotator Expertise in the Semi-Supervised Learning Scenario
This addresses the challenge of expensive and noisy labeling in domains like medical diagnosis, though it appears incremental as it combines existing semi-supervised and multi-annotator approaches.
The paper tackles the problem of learning from data with multiple noisy annotations and unlabeled data by proposing a probabilistic semi-supervised model that estimates true labels and annotator expertise, showing clear advantages over methods that ignore unlabeled data or multi-annotator information.
Learning algorithms normally assume that there is at most one annotation or label per data point. However, in some scenarios, such as medical diagnosis and on-line collaboration,multiple annotations may be available. In either case, obtaining labels for data points can be expensive and time-consuming (in some circumstances ground-truth may not exist). Semi-supervised learning approaches have shown that utilizing the unlabeled data is often beneficial in these cases. This paper presents a probabilistic semi-supervised model and algorithm that allows for learning from both unlabeled and labeled data in the presence of multiple annotators. We assume that it is known what annotator labeled which data points. The proposed approach produces annotator models that allow us to provide (1) estimates of the true label and (2) annotator variable expertise for both labeled and unlabeled data. We provide numerical comparisons under various scenarios and with respect to standard semi-supervised learning. Experiments showed that the presented approach provides clear advantages over multi-annotator methods that do not use the unlabeled data and over methods that do not use multi-labeler information.