SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation
This addresses the problem of scarce human evaluations for multilingual summarization by providing a large-scale dataset for researchers and practitioners, though it is incremental as it builds on existing evaluation frameworks.
The authors tackled the challenge of reliable automatic evaluation for summarization systems by introducing SEAHORSE, a multilingual dataset with 96K summaries and human ratings across 6 quality dimensions, covering 6 languages, 9 systems, and 4 datasets, and showed that metrics trained on it achieve strong performance on out-of-domain benchmarks.
Reliable automatic evaluation of summarization systems is challenging due to the multifaceted and subjective nature of the task. This is especially the case for languages other than English, where human evaluations are scarce. In this work, we introduce SEAHORSE, a dataset for multilingual, multifaceted summarization evaluation. SEAHORSE consists of 96K summaries with human ratings along 6 dimensions of text quality: comprehensibility, repetition, grammar, attribution, main ideas, and conciseness, covering 6 languages, 9 systems and 4 datasets. As a result of its size and scope, SEAHORSE can serve both as a benchmark to evaluate learnt metrics, as well as a large-scale resource for training such metrics. We show that metrics trained with SEAHORSE achieve strong performance on the out-of-domain meta-evaluation benchmarks TRUE (Honovich et al., 2022) and mFACE (Aharoni et al., 2022). We make the SEAHORSE dataset and metrics publicly available for future research on multilingual and multifaceted summarization evaluation.