Analytic Methods for Optimizing Realtime Crowdsourcing
This work addresses the scalability and optimization challenges in realtime crowdsourcing for applications like interactive systems, though it is incremental as it builds on existing retainer models with new analytical methods.
The paper tackled the problem of optimizing cost and performance in realtime crowdsourcing systems by analyzing the retainer model using queueing theory, resulting in an algorithm that minimizes requester cost while meeting performance requirements and demonstrating that precruited workers can start tasks within 500 milliseconds, achieving results below the one-second cognitive threshold.
Realtime crowdsourcing research has demonstrated that it is possible to recruit paid crowds within seconds by managing a small, fast-reacting worker pool. Realtime crowds enable crowd-powered systems that respond at interactive speeds: for example, cameras, robots and instant opinion polls. So far, these techniques have mainly been proof-of-concept prototypes: research has not yet attempted to understand how they might work at large scale or optimize their cost/performance trade-offs. In this paper, we use queueing theory to analyze the retainer model for realtime crowdsourcing, in particular its expected wait time and cost to requesters. We provide an algorithm that allows requesters to minimize their cost subject to performance requirements. We then propose and analyze three techniques to improve performance: push notifications, shared retainer pools, and precruitment, which involves recalling retainer workers before a task actually arrives. An experimental validation finds that precruited workers begin a task 500 milliseconds after it is posted, delivering results below the one-second cognitive threshold for an end-user to stay in flow.