The Effect of Class Imbalance and Order on Crowdsourced Relevance Judgments
This addresses a practical problem for crowdsourcing platforms and researchers by optimizing task design to enhance data quality, though it is incremental as it builds on existing work in human-computer interaction and data annotation.
The study investigated how class imbalance and presentation order affect crowd workers' efficiency and accuracy in relevance judgments, finding that showing relevant results before non-relevant ones significantly improves judgment quality.
In this paper we study the effect on crowd worker efficiency and effectiveness of the dominance of one class in the data they process. We aim at understanding if there is any positive or negative bias in workers seeing many negative examples in the identification of positive labels. To test our hypothesis, we design an experiment where crowd workers are asked to judge the relevance of documents presented in different orders. Our findings indicate that there is a significant improvement in the quality of relevance judgements when presenting relevant results before the non-relevant ones.