SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization
This addresses the need for efficient and scalable evaluation in multi-document summarization, though it is incremental as it builds on existing unsupervised metric approaches.
The paper tackles the problem of evaluating multi-document summaries without human references or annotations by proposing SUPERT, which uses semantic similarity with pseudo references and achieves 18-39% better correlation with human ratings compared to existing unsupervised metrics.
We study unsupervised multi-document summarization evaluation metrics, which require neither human-written reference summaries nor human annotations (e.g. preferences, ratings, etc.). We propose SUPERT, which rates the quality of a summary by measuring its semantic similarity with a pseudo reference summary, i.e. selected salient sentences from the source documents, using contextualized embeddings and soft token alignment techniques. Compared to the state-of-the-art unsupervised evaluation metrics, SUPERT correlates better with human ratings by 18-39%. Furthermore, we use SUPERT as rewards to guide a neural-based reinforcement learning summarizer, yielding favorable performance compared to the state-of-the-art unsupervised summarizers. All source code is available at https://github.com/yg211/acl20-ref-free-eval.