Large Language Models "Ad Referendum": How Good Are They at Machine Translation in the Legal Domain?
This incremental study addresses the problem of assessing translation quality in specialized legal domains for translators and researchers, suggesting LLMs may handle terminology better than traditional metrics indicate.
This study evaluated the machine translation quality of large language models (LLMs) compared to a traditional neural machine translation system in the legal domain, finding that while Google Translate performed better in automatic metrics, human evaluators rated LLMs like GPT-4 as comparable or slightly better for contextual adequacy and fluency.
This study evaluates the machine translation (MT) quality of two state-of-the-art large language models (LLMs) against a tradition-al neural machine translation (NMT) system across four language pairs in the legal domain. It combines automatic evaluation met-rics (AEMs) and human evaluation (HE) by professional transla-tors to assess translation ranking, fluency and adequacy. The re-sults indicate that while Google Translate generally outperforms LLMs in AEMs, human evaluators rate LLMs, especially GPT-4, comparably or slightly better in terms of producing contextually adequate and fluent translations. This discrepancy suggests LLMs' potential in handling specialized legal terminology and context, highlighting the importance of human evaluation methods in assessing MT quality. The study underscores the evolving capabil-ities of LLMs in specialized domains and calls for reevaluation of traditional AEMs to better capture the nuances of LLM-generated translations.