Measuring the Expertise of Workers for Crowdsourcing Applications
This work addresses the challenge of quality control in crowdsourcing platforms, which is crucial for companies relying on crowd-sourced tasks, though it appears incremental as it builds on existing methods like the Fagin distance.
The paper tackles the problem of evaluating worker expertise in crowdsourcing by proposing a new measure based on belief functions and four factors, and shows that fusing it with the Fagin distance improves results on a real audio quality assessment dataset.
Crowdsourcing platforms enable companies to propose tasks to a large crowd of users. The workers receive a compensation for their work according to the serious of the tasks they managed to accomplish. The evaluation of the quality of responses obtained from the crowd remains one of the most important problems in this context. Several methods have been proposed to estimate the expertise level of crowd workers. We propose an innovative measure of expertise assuming that we possess a dataset with an objective comparison of the items concerned. Our method is based on the definition of four factors with the theory of belief functions. We compare our method to the Fagin distance on a dataset from a real experiment, where users have to assess the quality of some audio recordings. Then, we propose to fuse both the Fagin distance and our expertise measure.