A Challenge Set Approach to Evaluating Machine Translation
This work provides a method for fine-grained error analysis in machine translation, helping researchers and developers identify specific linguistic challenges, though it is incremental as it builds on existing evaluation techniques.
The authors tackled the problem of evaluating machine translation systems by introducing a challenge set approach, which uses hand-designed sentences to probe specific structural divergences between languages, and applied it to analyze phrase-based and neural systems, revealing strengths and remaining weaknesses.
Neural machine translation represents an exciting leap forward in translation quality. But what longstanding weaknesses does it resolve, and which remain? We address these questions with a challenge set approach to translation evaluation and error analysis. A challenge set consists of a small set of sentences, each hand-designed to probe a system's capacity to bridge a particular structural divergence between languages. To exemplify this approach, we present an English-French challenge set, and use it to analyze phrase-based and neural systems. The resulting analysis provides not only a more fine-grained picture of the strengths of neural systems, but also insight into which linguistic phenomena remain out of reach.