Graph Mining Meets Crowdsourcing: Extracting Experts for Answer Aggregation
This addresses answer aggregation in crowdsourcing, particularly for scenarios with few experts among many non-experts, representing an incremental improvement over existing algorithms.
The paper tackles the problem of aggregating responses in crowdsourcing when experts are overwhelmed by non-experts, by introducing an 'expert core' concept and graph-mining algorithms to extract reliable experts, resulting in improved performance over state-of-the-art methods as demonstrated in computational experiments.
Aggregating responses from crowd workers is a fundamental task in the process of crowdsourcing. In cases where a few experts are overwhelmed by a large number of non-experts, most answer aggregation algorithms such as the majority voting fail to identify the correct answers. Therefore, it is crucial to extract reliable experts from the crowd workers. In this study, we introduce the notion of "expert core", which is a set of workers that is very unlikely to contain a non-expert. We design a graph-mining-based efficient algorithm that exactly computes the expert core. To answer the aggregation task, we propose two types of algorithms. The first one incorporates the expert core into existing answer aggregation algorithms such as the majority voting, whereas the second one utilizes information provided by the expert core extraction algorithm pertaining to the reliability of workers. We then give a theoretical justification for the first type of algorithm. Computational experiments using synthetic and real-world datasets demonstrate that our proposed answer aggregation algorithms outperform state-of-the-art algorithms.