Beyond AMT: An Analysis of Crowd Work Platforms
This study addresses the need for broader platform diversity in crowd work research to prevent bias and enrich future studies, though it is incremental in expanding analysis beyond AMT.
The paper tackles the over-reliance on Amazon Mechanical Turk in crowd work research by conducting a cross-platform analysis of seven platforms, highlighting how AMT's limitations may skew understanding and research directions.
While Amazon's Mechanical Turk (AMT) helped launch the paid crowd work industry eight years ago, many new vendors now offer a range of alternative models. Despite this, little crowd work research has explored other platforms. Such near-exclusive focus risks letting AMT's particular vagaries and limitations overly shape our understanding of crowd work and the research questions and directions being pursued. To address this, we present a cross-platform content analysis of seven crowd work platforms. We begin by reviewing how AMT assumptions and limitations have influenced prior research. Next, we formulate key criteria for characterizing and differentiating crowd work platforms. Our analysis of platforms contrasts them with AMT, informing both methodology of use and directions for future research. Our cross-platform analysis represents the only such study by researchers for researchers, intended to further enrich the diversity of research on crowd work and accelerate progress.