DMDTEval: An Evaluation and Analysis of LLMs on Disambiguation in Multi-domain Translation
This work addresses the challenge of improving LLM disambiguation for multi-domain translation, which is critical for enhancing translation accuracy in varied contexts, though it is incremental as it focuses on evaluation rather than a new method.
The paper tackles the problem of evaluating large language models (LLMs) on disambiguation in multi-domain translation, where word meanings vary across domains, by introducing DMDTEval, a systematic framework that includes a test set with ambiguous word annotations, disambiguation prompt strategies, and precise metrics, and conducts experiments across 4 language pairs and 13 domains to analyze LLM performance.
Currently, Large Language Models (LLMs) have achieved remarkable results in machine translation. However, their performance in multi-domain translation (MDT) is less satisfactory, the meanings of words can vary across different domains, highlighting the significant ambiguity inherent in MDT. Therefore, evaluating the disambiguation ability of LLMs in MDT, remains an open problem. To this end, we present an evaluation and analysis of LLMs on disambiguation in multi-domain translation (DMDTEval), our systematic evaluation framework consisting of three critical aspects: (1) we construct a translation test set with multi-domain ambiguous word annotation, (2) we curate a diverse set of disambiguation prompt strategies, and (3) we design precise disambiguation metrics, and study the efficacy of various prompt strategies on multiple state-of-the-art LLMs. We conduct comprehensive experiments across 4 language pairs and 13 domains, our extensive experiments reveal a number of crucial findings that we believe will pave the way and also facilitate further research in the critical area of improving the disambiguation of LLMs.