Quantifying and Avoiding Unfair Qualification Labour in Crowdsourcing
This addresses fairness issues for crowd workers, particularly in NLP and AI data collection, by proposing incremental improvements to reduce exploitation.
The paper tackles the problem of unfair qualification labor in crowdsourcing, where workers must complete substantial poorly paid tasks to qualify for better-paid work, estimating this burden at 2.25 months of full-time effort. It finds that alternatives can reduce this burden while maintaining high-quality data collection.
Extensive work has argued in favour of paying crowd workers a wage that is at least equivalent to the U.S. federal minimum wage. Meanwhile, research on collecting high quality annotations suggests using a qualification that requires workers to have previously completed a certain number of tasks. If most requesters who pay fairly require workers to have completed a large number of tasks already then workers need to complete a substantial amount of poorly paid work before they can earn a fair wage. Through analysis of worker discussions and guidance for researchers, we estimate that workers spend approximately 2.25 months of full time effort on poorly paid tasks in order to get the qualifications needed for better paid tasks. We discuss alternatives to this qualification and conduct a study of the correlation between qualifications and work quality on two NLP tasks. We find that it is possible to reduce the burden on workers while still collecting high quality data.