A comparison of translation performance between DeepL and Supertext
This addresses the need for context-sensitive evaluation methods in machine translation benchmarking for researchers and practitioners.
This study compared the translation performance of DeepL and Supertext on unsegmented texts, finding that while segment-level assessments showed no strong preference, document-level analysis revealed a preference for Supertext in three out of four language directions, indicating better consistency across longer texts.
As strong machine translation (MT) systems are increasingly based on large language models (LLMs), reliable quality benchmarking requires methods that capture their ability to leverage extended context. This study compares two commercial MT systems -- DeepL and Supertext -- by assessing their performance on unsegmented texts. We evaluate translation quality across four language directions with professional translators assessing segments with full document-level context. While segment-level assessments indicate no strong preference between the systems in most cases, document-level analysis reveals a preference for Supertext in three out of four language directions, suggesting superior consistency across longer texts. We advocate for more context-sensitive evaluation methodologies to ensure that MT quality assessments reflect real-world usability. We release all evaluation data and scripts for further analysis and reproduction at https://github.com/supertext/evaluation_deepl_supertext.