Incentivizing an Unknown Crowd
This addresses the challenge of incentivizing accurate labeling in crowdsourcing for applications like data annotation, though it appears incremental as it builds on existing EIWV frameworks with new robustness features.
The paper tackles the problem of sequential eliciting information without verification (EIWV) for a heterogeneous and unknown crowd in crowdsourcing labeling, proposing a reinforcement learning-based approach that dynamically decides oracle calls to gain robustness against irrationality and collusion, with extensive experiments showing its advantage on large-scale real datasets.
Motivated by the common strategic activities in crowdsourcing labeling, we study the problem of sequential eliciting information without verification (EIWV) for workers with a heterogeneous and unknown crowd. We propose a reinforcement learning-based approach that is effective against a wide range of settings including potential irrationality and collusion among workers. With the aid of a costly oracle and the inference method, our approach dynamically decides the oracle calls and gains robustness even under the presence of frequent collusion activities. Extensive experiments show the advantage of our approach. Our results also present the first comprehensive experiments of EIWV on large-scale real datasets and the first thorough study of the effects of environmental variables.