Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models
This work addresses the need for more accurate and human-like evaluation methods in machine translation, particularly for segment-level assessment, though it is incremental as it builds on existing prompting techniques.
The paper tackles the problem of poor segment-level performance in large language models (LLMs) for machine translation quality assessment by proposing Error Analysis Prompting (EAPrompt), which combines Chain-of-Thoughts and Error Analysis to emulate human evaluation frameworks, achieving explainable and reliable evaluations at both system and segment levels as validated on the WMT22 metrics shared task.
Generative large language models (LLMs), e.g., ChatGPT, have demonstrated remarkable proficiency across several NLP tasks, such as machine translation, text summarization. Recent research (Kocmi and Federmann, 2023) has shown that utilizing LLMs for assessing the quality of machine translation (MT) achieves state-of-the-art performance at the system level but \textit{performs poorly at the segment level}. To further improve the performance of LLMs on MT quality assessment, we investigate several prompting designs, and propose a new prompting method called \textbf{\texttt{Error Analysis Prompting}} (EAPrompt) by combining Chain-of-Thoughts (Wei et al., 2022) and Error Analysis (Lu et al., 2023). This technique emulates the commonly accepted human evaluation framework - Multidimensional Quality Metrics (MQM, Freitag et al. (2021)) and \textit{produces explainable and reliable MT evaluations at both the system and segment level}. Experimental Results from the WMT22 metrics shared task validate the effectiveness of EAPrompt on various LLMs, with different structures. Further analysis confirms that EAPrompt effectively distinguishes major errors from minor ones, while also sharing a similar distribution of the number of errors with MQM. These findings highlight the potential of EAPrompt as a human-like evaluator prompting technique for MT evaluation.