A Study of Unsupervised Adaptive Crowdsourcing
Provides theoretical analysis for unsupervised crowdsourcing, which is relevant for systems relying on majority voting without ground truth.
The paper studies unsupervised crowdsourcing performance by correlating user responses with the majority, finding that overall crowd reliability is a key factor in both sequential and single-assignment settings.
We consider unsupervised crowdsourcing performance based on the model wherein the responses of end-users are essentially rated according to how their responses correlate with the majority of other responses to the same subtasks/questions. In one setting, we consider an independent sequence of identically distributed crowdsourcing assignments (meta-tasks), while in the other we consider a single assignment with a large number of component subtasks. Both problems yield intuitive results in which the overall reliability of the crowd is a factor.