MAS-LitEval : Multi-Agent System for Literary Translation Quality Assessment
This provides a scalable, nuanced framework for Translation Quality Assessment, offering a practical tool for translators and researchers, though it is incremental as it builds on existing LLM methods.
The paper tackled the problem of assessing literary translation quality, which traditional metrics fail to capture, by proposing MAS-LitEval, a multi-agent system using LLMs that outperformed traditional metrics with scores up to 0.890 in capturing nuances.
Literary translation requires preserving cultural nuances and stylistic elements, which traditional metrics like BLEU and METEOR fail to assess due to their focus on lexical overlap. This oversight neglects the narrative consistency and stylistic fidelity that are crucial for literary works. To address this, we propose MAS-LitEval, a multi-agent system using Large Language Models (LLMs) to evaluate translations based on terminology, narrative, and style. We tested MAS-LitEval on translations of The Little Prince and A Connecticut Yankee in King Arthur's Court, generated by various LLMs, and compared it to traditional metrics. \textbf{MAS-LitEval} outperformed these metrics, with top models scoring up to 0.890 in capturing literary nuances. This work introduces a scalable, nuanced framework for Translation Quality Assessment (TQA), offering a practical tool for translators and researchers.