Learning to Incentivize: Eliciting Effort via Output Agreement
This work addresses the challenge of eliciting effort in crowdsourcing without verification, which is incremental as it extends existing output agreement mechanisms to incorporate effort incentives.
The paper tackles the problem of using output agreement mechanisms to incentivize both truthful answers and effort from crowdsourced workers with heterogeneous effort costs, deriving the optimal reward level when the cost distribution is known and developing sequential mechanisms to approximate it when unknown.
In crowdsourcing when there is a lack of verification for contributed answers, output agreement mechanisms are often used to incentivize participants to provide truthful answers when the correct answer is hold by the majority. In this paper, we focus on using output agreement mechanisms to elicit effort, in addition to eliciting truthful answers, from a population of workers. We consider a setting where workers have heterogeneous cost of effort exertion and examine the data requester's problem of deciding the reward level in output agreement for optimal elicitation. In particular, when the requester knows the cost distribution, we derive the optimal reward level for output agreement mechanisms. This is achieved by first characterizing Bayesian Nash equilibria of output agreement mechanisms for a given reward level. When the requester does not know the cost distribution, we develop sequential mechanisms that combine learning the cost distribution with incentivizing effort exertion to approximately determine the optimal reward level.