Investigating Failures of Automatic Translation in the Case of Unambiguous Gender
This addresses a systemic error in machine translation that affects translation quality for languages with grammatical gender, though it is incremental as it focuses on a specific class of errors.
The paper tackles the problem of transformer-based neural machine translation models failing to correctly assign grammatical gender to occupation nouns even when context provides unambiguous evidence, and introduces a dataset for evaluating this error across 20 languages.
Transformer based models are the modern work horses for neural machine translation (NMT), reaching state of the art across several benchmarks. Despite their impressive accuracy, we observe a systemic and rudimentary class of errors made by transformer based models with regards to translating from a language that doesn't mark gender on nouns into others that do. We find that even when the surrounding context provides unambiguous evidence of the appropriate grammatical gender marking, no transformer based model we tested was able to accurately gender occupation nouns systematically. We release an evaluation scheme and dataset for measuring the ability of transformer based NMT models to translate gender morphology correctly in unambiguous contexts across syntactically diverse sentences. Our dataset translates from an English source into 20 languages from several different language families. With the availability of this dataset, our hope is that the NMT community can iterate on solutions for this class of especially egregious errors.