Multi-object Classification via Crowdsourcing with a Reject Option
This work addresses the challenge of improving accuracy in crowdsourced classification tasks, particularly for scenarios like mismatched speech transcription, but it is incremental as it builds on existing aggregation methods.
The paper tackles the problem of multi-class classification via crowdsourcing when workers can skip tasks they cannot perform reliably, proposing a weighted majority voting rule with optimized weights to maximize classification performance, and shows improved results over conventional majority voting in simulations.
Consider designing an effective crowdsourcing system for an $M$-ary classification task. Crowd workers complete simple binary microtasks whose results are aggregated to give the final result. We consider the novel scenario where workers have a reject option so they may skip microtasks when they are unable or choose not to respond. For example, in mismatched speech transcription, workers who do not know the language may not be able to respond to microtasks focused on phonological dimensions outside their categorical perception. We present an aggregation approach using a weighted majority voting rule, where each worker's response is assigned an optimized weight to maximize the crowd's classification performance. We evaluate system performance in both exact and asymptotic forms. Further, we consider the setting where there may be a set of greedy workers that complete microtasks even when they are unable to perform it reliably. We consider an oblivious and an expurgation strategy to deal with greedy workers, developing an algorithm to adaptively switch between the two based on the estimated fraction of greedy workers in the anonymous crowd. Simulation results show improved performance compared with conventional majority voting.