Hindsight Quality Prediction Experiments in Multi-Candidate Human-Post-Edited Machine Translation
This research is significant for developers and researchers in machine translation, as it evaluates how the integration of Large Language Models affects existing quality prediction paradigms, providing insights into their altered reliability.
This paper investigates source-side difficulty prediction and candidate-side quality estimation (QE) for machine translation (MT) quality, using a dataset of over 6,000 English source segments with nine translation hypotheses from various MT systems and LLMs. The study found that the shift to LLMs impacts the reliability of established quality prediction methods while improving document-level translation.
This paper investigates two complementary paradigms for predicting machine translation (MT) quality: source-side difficulty prediction and candidate-side quality estimation (QE). The rapid adoption of Large Language Models (LLMs) into MT workflows is reshaping the research landscape, yet its impact on established quality prediction paradigms remains underexplored. We study this issue through a series of "hindsight" experiments on a unique, multi-candidate dataset resulting from a genuine MT post-editing (MTPE) project. The dataset consists of over 6,000 English source segments with nine translation hypotheses from a diverse set of traditional neural MT systems and advanced LLMs, all evaluated against a single, final human post-edited reference. Using Kendall's rank correlation, we assess the predictive power of source-side difficulty metrics, candidate-side QE models and position heuristics against two gold-standard scores: TER (as a proxy for post-editing effort) and COMET (as a proxy for human judgment). Our findings highlight that the architectural shift towards LLMs alters the reliability of established quality prediction methods while simultaneously mitigating previous challenges in document-level translation.