SueNes: A Weakly Supervised Approach to Evaluating Single-Document Summarization via Negative Sampling
This addresses the need for more efficient and semantically-aware evaluation metrics in summarization, though it is incremental as it builds on existing weakly supervised methods.
The paper tackles the problem of evaluating single-document summarization without costly reference summaries by proposing SueNes, a weakly supervised approach that uses corrupted reference summaries for training. In cross-domain tests, it outperforms baselines with promising improvements and shows a great advantage in gauging linguistic qualities.
Canonical automatic summary evaluation metrics, such as ROUGE, focus on lexical similarity which cannot well capture semantics nor linguistic quality and require a reference summary which is costly to obtain. Recently, there have been a growing number of efforts to alleviate either or both of the two drawbacks. In this paper, we present a proof-of-concept study to a weakly supervised summary evaluation approach without the presence of reference summaries. Massive data in existing summarization datasets are transformed for training by pairing documents with corrupted reference summaries. In cross-domain tests, our strategy outperforms baselines with promising improvements, and show a great advantage in gauging linguistic qualities over all metrics.