What's the Difference Between Professional Human and Machine Translation? A Blind Multi-language Study on Domain-specific MT
This addresses the problem of evaluating translation quality for users in domain-specific contexts, but it is incremental as it builds on existing MT error analysis.
The study compared professional human translation (HT) and machine translation (MT) by conducting a blind evaluation where translators flagged errors and post-edited interleaved documents, finding that post-editing effort for MT was higher in only two out of three language pairs and error types like wrong terminology were similar in HT.
Machine translation (MT) has been shown to produce a number of errors that require human post-editing, but the extent to which professional human translation (HT) contains such errors has not yet been compared to MT. We compile pre-translated documents in which MT and HT are interleaved, and ask professional translators to flag errors and post-edit these documents in a blind evaluation. We find that the post-editing effort for MT segments is only higher in two out of three language pairs, and that the number of segments with wrong terminology, omissions, and typographical problems is similar in HT.