Does Summary Evaluation Survive Translation to Other Languages?
This addresses the problem of costly dataset creation for summarization evaluation, showing limited reuse potential for researchers in NLP.
The study investigated whether machine-translated summarization datasets can be reliably used across languages by translating the English SummEval dataset to seven languages and comparing automatic evaluation measures, finding that most methods are not statistically equivalent across translations.
The creation of a quality summarization dataset is an expensive, time-consuming effort, requiring the production and evaluation of summaries by both trained humans and machines. If such effort is made in one language, it would be beneficial to be able to use it in other languages without repeating human annotations. To investigate how much we can trust machine translation of such a dataset, we translate the English dataset SummEval to seven languages and compare performance across automatic evaluation measures. We explore equivalence testing as the appropriate statistical paradigm for evaluating correlations between human and automated scoring of summaries. While we find some potential for dataset reuse in languages similar to the source, most summary evaluation methods are not found to be statistically equivalent across translations.