Quality Enhancement by Weighted Rank Aggregation of Crowd Opinion
This work addresses the challenge of enhancing annotation quality in crowdsourcing platforms, which is incremental as it builds on existing opinion aggregation models by incorporating multiple features.
The paper tackles the problem of improving ground truth prediction in crowdsourcing by aggregating multiple rankings of annotators based on various features, rather than relying on a single feature. It proposes a novel weighted rank aggregation method and demonstrates its efficacy on both an artificial dataset and an Amazon Mechanical Turk dataset.
Expertise of annotators has a major role in crowdsourcing based opinion aggregation models. In such frameworks, accuracy and biasness of annotators are occasionally taken as important features and based on them priority of the annotators are assigned. But instead of relying on a single feature, multiple features can be considered and separate rankings can be produced to judge the annotators properly. Finally, the aggregation of those rankings with perfect weightage can be done with an aim to produce better ground truth prediction. Here, we propose a novel weighted rank aggregation method and its efficacy with respect to other existing approaches is shown on artificial dataset. The effectiveness of weighted rank aggregation to enhance quality prediction is also shown by applying it on an Amazon Mechanical Turk (AMT) dataset.