A 2-step Framework for Automated Literary Translation Evaluation: Its Promises and Pitfalls
This work addresses the need for fine-grained and culturally sensitive evaluation methods in literary machine translation, though it is incremental as it builds on existing frameworks.
The authors tackled the problem of evaluating literary machine translation from English to Korean by proposing a two-stage pipeline, which achieved higher correlation with human judgment than traditional metrics but still fell short of inter-human agreement, particularly in areas like Korean honorifics.
In this work, we propose and evaluate the feasibility of a two-stage pipeline to evaluate literary machine translation, in a fine-grained manner, from English to Korean. The results show that our framework provides fine-grained, interpretable metrics suited for literary translation and obtains a higher correlation with human judgment than traditional machine translation metrics. Nonetheless, it still fails to match inter-human agreement, especially in metrics like Korean Honorifics. We also observe that LLMs tend to favor translations generated by other LLMs, and we highlight the necessity of developing more sophisticated evaluation methods to ensure accurate and culturally sensitive machine translation of literary works.