Using Mechanical Turk to Build Machine Translation Evaluation Sets
This addresses the problem of expensive test set creation for machine translation researchers and practitioners, offering a cost-effective alternative.
The paper tackled the high cost of building machine translation test sets by using Amazon's Mechanical Turk, finding it reduces costs significantly while producing test sets that yield the same performance conclusions as professional ones.
Building machine translation (MT) test sets is a relatively expensive task. As MT becomes increasingly desired for more and more language pairs and more and more domains, it becomes necessary to build test sets for each case. In this paper, we investigate using Amazon's Mechanical Turk (MTurk) to make MT test sets cheaply. We find that MTurk can be used to make test sets much cheaper than professionally-produced test sets. More importantly, in experiments with multiple MT systems, we find that the MTurk-produced test sets yield essentially the same conclusions regarding system performance as the professionally-produced test sets yield.