Automatically Extracting Challenge Sets for Non local Phenomena in Neural Machine Translation
This work addresses the problem of evaluating machine translation systems on rare syntactic phenomena for researchers and practitioners, though it is incremental as it builds on existing challenge set methodologies.
The authors tackled the challenge of evaluating neural machine translation models on long-distance dependencies by proposing an automatic method to extract large challenge sets, resulting in significantly larger sets for English-German and German-English that enable reliable automatic evaluation.
We show that the state of the art Transformer Machine Translation (MT) model is not biased towards monotonic reordering (unlike previous recurrent neural network models), but that nevertheless, long-distance dependencies remain a challenge for the model. Since most dependencies are short-distance, common evaluation metrics will be little influenced by how well systems perform on them. We, therefore, propose an automatic approach for extracting challenge sets replete with long-distance dependencies and argue that evaluation using this methodology provides a complementary perspective on system performance. To support our claim, we compile challenge sets for English-German and German-English, which are much larger than any previously released challenge set for MT. The extracted sets are large enough to allow reliable automatic evaluation, which makes the proposed approach a scalable and practical solution for evaluating MT performance on the long-tail of syntactic phenomena.