Improving Label Quality by Jointly Modeling Items and Annotators
This work addresses the challenge of improving label quality for machine learning tasks, particularly in scenarios with noisy annotations, but it appears incremental as it builds on existing models like David and Skene.
The paper tackles the problem of learning ground truth labels from noisy annotators by proposing a fully Bayesian framework that jointly models items and annotators, and it shows improved performance in supervised learning experiments compared to a baseline and a state-of-the-art model.
We propose a fully Bayesian framework for learning ground truth labels from noisy annotators. Our framework ensures scalability by factoring a generative, Bayesian soft clustering model over label distributions into the classic David and Skene joint annotator-data model. Earlier research along these lines has neither fully incorporated label distributions nor explored clustering by annotators only or data only. Our framework incorporates all of these properties as: (1) a graphical model designed to provide better ground truth estimates of annotator responses as input to \emph{any} black box supervised learning algorithm, and (2) a standalone neural model whose internal structure captures many of the properties of the graphical model. We conduct supervised learning experiments using both models and compare them to the performance of one baseline and a state-of-the-art model.