Avoiding Imposters and Delinquents: Adversarial Crowdsourcing and Peer Prediction
This addresses the challenge of reliable quality assessment in crowdsourcing and peer prediction systems, such as online peer grading, with potential applications in scalable content moderation and educational platforms.
The paper tackles the problem of curating high-quality items from crowdsourced ratings in the presence of adversarial workers, showing that it is possible to achieve this with manager and worker efforts that do not scale with the number of items or workers, requiring only $ ilde{O}\Big( rac{1}{βα^3ε^4}\Big)$ ratings per worker and $ ilde{O}\Big( rac{1}{βε^2}\Big)$ ratings by the manager.
We consider a crowdsourcing model in which $n$ workers are asked to rate the quality of $n$ items previously generated by other workers. An unknown set of $αn$ workers generate reliable ratings, while the remaining workers may behave arbitrarily and possibly adversarially. The manager of the experiment can also manually evaluate the quality of a small number of items, and wishes to curate together almost all of the high-quality items with at most an $ε$ fraction of low-quality items. Perhaps surprisingly, we show that this is possible with an amount of work required of the manager, and each worker, that does not scale with $n$: the dataset can be curated with $\tilde{O}\Big(\frac{1}{βα^3ε^4}\Big)$ ratings per worker, and $\tilde{O}\Big(\frac{1}{βε^2}\Big)$ ratings by the manager, where $β$ is the fraction of high-quality items. Our results extend to the more general setting of peer prediction, including peer grading in online classrooms.