Understanding Translationese in Cross-Lingual Summarization
This work addresses a subtle but important issue in cross-lingual summarization for researchers and practitioners, highlighting dataset biases and evaluation challenges, though it is incremental in nature.
The paper investigates how translationese, the distinct characteristics of translated text, affects cross-lingual summarization (CLS) by analyzing its impact on dataset construction, model evaluation, and performance, finding that it can cause discrepancies in human vs. automatic evaluation and harm real-world applicability, but remains useful for low-resource languages under specific strategies.
Given a document in a source language, cross-lingual summarization (CLS) aims at generating a concise summary in a different target language. Unlike monolingual summarization (MS), naturally occurring source-language documents paired with target-language summaries are rare. To collect large-scale CLS data, existing datasets typically involve translation in their creation. However, the translated text is distinguished from the text originally written in that language, i.e., translationese. In this paper, we first confirm that different approaches of constructing CLS datasets will lead to different degrees of translationese. Then we systematically investigate how translationese affects CLS model evaluation and performance when it appears in source documents or target summaries. In detail, we find that (1) the translationese in documents or summaries of test sets might lead to the discrepancy between human judgment and automatic evaluation; (2) the translationese in training sets would harm model performance in real-world applications; (3) though machine-translated documents involve translationese, they are very useful for building CLS systems on low-resource languages under specific training strategies. Lastly, we give suggestions for future CLS research including dataset and model developments. We hope that our work could let researchers notice the phenomenon of translationese in CLS and take it into account in the future.