Evaluating LLMs on Chinese Idiom Translation
This addresses the challenge of accurate idiom translation for Chinese language users and NLP applications, but it is incremental as it focuses on evaluation and improvement within an existing framework.
The paper tackled the problem of Chinese idiom translation by evaluating nine modern systems, including GPT-4o and Google Translate, and found they fail frequently, with the best-performing GPT-4 making errors in 28% of cases; it also developed improved models achieving an F1 score of 0.68 for error detection.
Idioms, whose figurative meanings usually differ from their literal interpretations, are common in everyday language, especially in Chinese, where they often contain historical references and follow specific structural patterns. Despite recent progress in machine translation with large language models, little is known about Chinese idiom translation. In this work, we introduce IdiomEval, a framework with a comprehensive error taxonomy for Chinese idiom translation. We annotate 900 translation pairs from nine modern systems, including GPT-4o and Google Translate, across four domains: web, news, Wikipedia, and social media. We find these systems fail at idiom translation, producing incorrect, literal, partial, or even missing translations. The best-performing system, GPT-4, makes errors in 28% of cases. We also find that existing evaluation metrics measure idiom quality poorly with Pearson correlation below 0.48 with human ratings. We thus develop improved models that achieve F$_1$ scores of 0.68 for detecting idiom translation errors.