Multi-annotator Deep Learning: A Probabilistic Framework for Classification
This addresses the challenge of improving classification accuracy in scenarios with noisy crowd-sourced annotations, though it is an incremental advancement in multi-annotator learning methods.
The paper tackles the problem of noisy class labels from multiple error-prone annotators in deep learning classification tasks by introducing a probabilistic framework called multi-annotator deep learning (MaDL), which jointly trains ground truth and annotator performance models to achieve state-of-the-art performance and robustness against correlated annotators.
Solving complex classification tasks using deep neural networks typically requires large amounts of annotated data. However, corresponding class labels are noisy when provided by error-prone annotators, e.g., crowdworkers. Training standard deep neural networks leads to subpar performances in such multi-annotator supervised learning settings. We address this issue by presenting a probabilistic training framework named multi-annotator deep learning (MaDL). A downstream ground truth and an annotator performance model are jointly trained in an end-to-end learning approach. The ground truth model learns to predict instances' true class labels, while the annotator performance model infers probabilistic estimates of annotators' performances. A modular network architecture enables us to make varying assumptions regarding annotators' performances, e.g., an optional class or instance dependency. Further, we learn annotator embeddings to estimate annotators' densities within a latent space as proxies of their potentially correlated annotations. Together with a weighted loss function, we improve the learning from correlated annotation patterns. In a comprehensive evaluation, we examine three research questions about multi-annotator supervised learning. Our findings show MaDL's state-of-the-art performance and robustness against many correlated, spamming annotators.