A Large-Scale Test Set for the Evaluation of Context-Aware Pronoun Translation in Neural Machine Translation
This work addresses the problem of accurately evaluating pronoun translation for machine translation researchers, though it is incremental as it builds on existing context-aware models.
The authors tackled the challenge of evaluating context-aware pronoun translation in neural machine translation by creating a large-scale contrastive test set focused on pronouns, and demonstrated that context-aware models significantly outperform baselines on this test set despite only moderate BLEU score improvements.
The translation of pronouns presents a special challenge to machine translation to this day, since it often requires context outside the current sentence. Recent work on models that have access to information across sentence boundaries has seen only moderate improvements in terms of automatic evaluation metrics such as BLEU. However, metrics that quantify the overall translation quality are ill-equipped to measure gains from additional context. We argue that a different kind of evaluation is needed to assess how well models translate inter-sentential phenomena such as pronouns. This paper therefore presents a test suite of contrastive translations focused specifically on the translation of pronouns. Furthermore, we perform experiments with several context-aware models. We show that, while gains in BLEU are moderate for those systems, they outperform baselines by a large margin in terms of accuracy on our contrastive test set. Our experiments also show the effectiveness of parameter tying for multi-encoder architectures.