Incorporating Worker Perspectives into MTurk Annotation Practices for NLP
This addresses fairness and data quality issues for NLP researchers and MTurk workers, but it is incremental as it builds on existing critiques without introducing new methods.
The paper tackled the problem of data collection practices on Amazon Mechanical Turk for NLP by surveying workers, finding that their preferences often conflict with common researcher practices, such as preferring reliable payments over high uncertain ones and viewing some quality control methods as biased.
Current practices regarding data collection for natural language processing on Amazon Mechanical Turk (MTurk) often rely on a combination of studies on data quality and heuristics shared among NLP researchers. However, without considering the perspectives of MTurk workers, these approaches are susceptible to issues regarding workers' rights and poor response quality. We conducted a critical literature review and a survey of MTurk workers aimed at addressing open questions regarding best practices for fair payment, worker privacy, data quality, and considering worker incentives. We found that worker preferences are often at odds with received wisdom among NLP researchers. Surveyed workers preferred reliable, reasonable payments over uncertain, very high payments; reported frequently lying on demographic questions; and expressed frustration at having work rejected with no explanation. We also found that workers view some quality control methods, such as requiring minimum response times or Master's qualifications, as biased and largely ineffective. Based on the survey results, we provide recommendations on how future NLP studies may better account for MTurk workers' experiences in order to respect workers' rights and improve data quality.