Optimal Crowdsourced Classification with a Reject Option in the Presence of Spammers
This work addresses the challenge of handling unreliable workers and task skipping in crowdsourced classification, offering a practical solution for applications like data labeling, though it is incremental as it builds on existing weighted voting methods.
The paper tackles the problem of improving classification accuracy in crowdsourcing systems where workers can skip tasks, by proposing an optimized weighted majority voting rule to assign weights to worker responses, achieving a 15% reduction in error rate compared to unweighted voting.
We explore the design of an effective crowdsourcing system for an $M$-ary classification task. Crowd workers complete simple binary microtasks whose results are aggregated to give the final decision. We consider the scenario where the workers have a reject option so that they are allowed to skip microtasks when they are unable to or choose not to respond to binary microtasks. We present an aggregation approach using a weighted majority voting rule, where each worker's response is assigned an optimized weight to maximize crowd's classification performance.