CrowdHub: Extending crowdsourcing platforms for the controlled evaluation of tasks designs
This tool addresses bias and efficiency issues for researchers using crowdsourcing platforms, though it is incremental as it builds on existing platforms.
The authors tackled the problem of experimental bias in crowdsourcing task evaluations by developing CrowdHub, a tool that automates and controls task execution, which reduced bias and increased dataset utility by up to 38% compared to uncontrolled settings.
We present CrowdHub, a tool for running systematic evaluations of task designs on top of crowdsourcing platforms. The goal is to support the evaluation process, avoiding potential experimental biases that, according to our empirical studies, can amount to 38% loss in the utility of the collected dataset in uncontrolled settings. Using CrowdHub, researchers can map their experimental design and automate the complex process of managing task execution over time while controlling for returning workers and crowd demographics, thus reducing bias, increasing utility of collected data, and making more efficient use of a limited pool of subjects.