Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering
This addresses the challenge of evaluating meaning preservation in long, complex translations for researchers and practitioners in machine translation, though it is an incremental improvement over existing extrinsic methods.
The paper tackles the problem of evaluating paragraph-level machine translation by introducing TREQA, a framework that uses question-answering to assess translation quality, showing it is competitive with or outperforms state-of-the-art metrics in ranking translations for literary texts.
Despite the steady progress in machine translation evaluation, existing automatic metrics struggle to capture how well meaning is preserved beyond sentence boundaries. We posit that reliance on a single intrinsic quality score, trained to mimic human judgments, might be insufficient for evaluating translations of long, complex passages, and a more ``pragmatic'' approach that assesses how accurately key information is conveyed by a translation in context is needed. We introduce TREQA (Translation Evaluation via Question-Answering), a framework that extrinsically evaluates translation quality by assessing how accurately candidate translations answer reading comprehension questions that target key information in the original source or reference texts. In challenging domains that require long-range understanding, such as literary texts, we show that TREQA is competitive with and, in some cases, outperforms state-of-the-art neural and LLM-based metrics in ranking alternative paragraph-level translations, despite never being explicitly optimized to correlate with human judgments. Furthermore, the generated questions and answers offer interpretability: empirical analysis shows that they effectively target translation errors identified by experts in evaluated datasets. Our code is available at https://github.com/deep-spin/treqa