Evaluating Optimal Reference Translations
This addresses the challenge of accurate evaluation for researchers and developers in machine translation, though it is incremental as it builds on existing reference translation practices.
The paper tackles the problem of unreliable reference translations in high-resource machine translation evaluation by proposing a methodology for creating 'optimal reference translations' at the document level, which significantly increases quality compared to standard references.
The overall translation quality reached by current machine translation (MT) systems for high-resourced language pairs is remarkably good. Standard methods of evaluation are not suitable nor intended to uncover the many translation errors and quality deficiencies that still persist. Furthermore, the quality of standard reference translations is commonly questioned and comparable quality levels have been reached by MT alone in several language pairs. Navigating further research in these high-resource settings is thus difficult. In this article, we propose a methodology for creating more reliable document-level human reference translations, called "optimal reference translations," with the simple aim to raise the bar of what should be deemed "human translation quality." We evaluate the obtained document-level optimal reference translations in comparison with "standard" ones, confirming a significant quality increase and also documenting the relationship between evaluation and translation editing.