Inferring ground truth from multi-annotator ordinal data: a probabilistic approach
This work addresses the challenge of improving data quality in crowdsourced ordinal annotation tasks, such as query-URL relevance scoring, by providing a more robust method for aggregating noisy labels, though it is incremental as it builds on existing models for binary/categorical data.
The authors tackled the problem of inferring ground truth from noisy ordinal labels provided by multiple annotators with varying expertise, proposing a probabilistic model that accounts for instance difficulty and annotator expertise. Their model performed better or as well as existing state-of-the-art methods and was more resistant to spammy annotators compared to baselines like mean, median, and majority vote.
A popular approach for large scale data annotation tasks is crowdsourcing, wherein each data point is labeled by multiple noisy annotators. We consider the problem of inferring ground truth from noisy ordinal labels obtained from multiple annotators of varying and unknown expertise levels. Annotation models for ordinal data have been proposed mostly as extensions of their binary/categorical counterparts and have received little attention in the crowdsourcing literature. We propose a new model for crowdsourced ordinal data that accounts for instance difficulty as well as annotator expertise, and derive a variational Bayesian inference algorithm for parameter estimation. We analyze the ordinal extensions of several state-of-the-art annotator models for binary/categorical labels and evaluate the performance of all the models on two real world datasets containing ordinal query-URL relevance scores, collected through Amazon's Mechanical Turk. Our results indicate that the proposed model performs better or as well as existing state-of-the-art methods and is more resistant to `spammy' annotators (i.e., annotators who assign labels randomly without actually looking at the instance) than popular baselines such as mean, median, and majority vote which do not account for annotator expertise.