Benchmarking Machine Translation with Cultural Awareness
This work addresses the problem of assessing cultural awareness in machine translation for researchers and developers, though it is incremental as it builds on existing translation paradigms with new data and metrics.
The paper tackled the challenge of evaluating machine translation systems on culture-specific items by introducing a new annotated parallel corpus and evaluation metrics, finding that large language models outperform neural machine translation in leveraging external cultural knowledge for translating items lacking target-culture equivalents.
Translating culture-related content is vital for effective cross-cultural communication. However, many culture-specific items (CSIs) often lack viable translations across languages, making it challenging to collect high-quality, diverse parallel corpora with CSI annotations. This difficulty hinders the analysis of cultural awareness of machine translation (MT) systems, including traditional neural MT and the emerging MT paradigm using large language models (LLM). To address this gap, we introduce a novel parallel corpus, enriched with CSI annotations in 6 language pairs for investigating Culturally-Aware Machine Translation--CAMT. Furthermore, we design two evaluation metrics to assess CSI translations, focusing on their pragmatic translation quality. Our findings show the superior ability of LLMs over neural MTs in leveraging external cultural knowledge for translating CSIs, especially those lacking translations in the target culture.