Knowledge-Prompted Estimator: A Novel Approach to Explainable Machine Translation Assessment
This work addresses the need for explainable, segment-level assessment in machine translation, offering an incremental improvement over existing methods.
The paper tackled the problem of segment-level machine translation quality estimation by introducing the Knowledge-Prompted Estimator (KPE), a Chain-of-Thought prompting method that combines three one-step techniques, achieving enhanced performance compared to previous models and approaches.
Cross-lingual Machine Translation (MT) quality estimation plays a crucial role in evaluating translation performance. GEMBA, the first MT quality assessment metric based on Large Language Models (LLMs), employs one-step prompting to achieve state-of-the-art (SOTA) in system-level MT quality estimation; however, it lacks segment-level analysis. In contrast, Chain-of-Thought (CoT) prompting outperforms one-step prompting by offering improved reasoning and explainability. In this paper, we introduce Knowledge-Prompted Estimator (KPE), a CoT prompting method that combines three one-step prompting techniques, including perplexity, token-level similarity, and sentence-level similarity. This method attains enhanced performance for segment-level estimation compared with previous deep learning models and one-step prompting approaches. Furthermore, supplementary experiments on word-level visualized alignment demonstrate that our KPE method significantly improves token alignment compared with earlier models and provides better interpretability for MT quality estimation. Code will be released upon publication.