Evaluating Large Language Models on Urdu Idiom Translation
This work addresses idiomatic translation for low-resource Urdu, which is an incremental contribution to machine translation research.
The authors tackled the challenge of idiomatic translation for low-resource Urdu by creating the first evaluation datasets for Urdu to English idiomatic translation and evaluating multiple LLMs and NMT systems. They found that prompt engineering improves idiomatic translation, with Native Urdu inputs yielding more accurate translations than Roman Urdu.
Idiomatic translation remains a significant challenge in machine translation, especially for low resource languages such as Urdu, and has received limited prior attention. To advance research in this area, we introduce the first evaluation datasets for Urdu to English idiomatic translation, covering both Native Urdu and Roman Urdu scripts and annotated with gold-standard English equivalents. We evaluate multiple open-source Large Language Models (LLMs) and Neural Machine Translation (NMT) systems on this task, focusing on their ability to preserve idiomatic and cultural meaning. Automatic metrics including BLEU, BERTScore, COMET, and XCOMET are used to assess translation quality. Our findings indicate that prompt engineering enhances idiomatic translation compared to direct translation, though performance differences among prompt types are relatively minor. Moreover, cross script comparisons reveal that text representation substantially affects translation quality, with Native Urdu inputs producing more accurate idiomatic translations than Roman Urdu.