DocAsRef: An Empirical Study on Repurposing Reference-Based Summary Quality Metrics Reference-Freely
This work addresses the need for automated summary assessment without human references, offering a practical improvement for NLP researchers and practitioners, though it is incremental as it adapts existing metrics.
The study tackled the limitation of reference-based summary quality metrics by adapting their comparison methodologies to work without human references, using the source document instead. The repurposed zero-shot BERTScore with DeBERTa-large-MNLI outperformed its reference-based version and most existing reference-free metrics, achieving competitive results with GPT-3.5-based evaluators.
Automated summary quality assessment falls into two categories: reference-based and reference-free. Reference-based metrics, historically deemed more accurate due to the additional information provided by human-written references, are limited by their reliance on human input. In this paper, we hypothesize that the comparison methodologies used by some reference-based metrics to evaluate a system summary against its corresponding reference can be effectively adapted to assess it against its source document, thereby transforming these metrics into reference-free ones. Experimental results support this hypothesis. After being repurposed reference-freely, the zero-shot BERTScore using the pretrained DeBERTa-large-MNLI model of <0.5B parameters consistently outperforms its original reference-based version across various aspects on the SummEval and Newsroom datasets. It also excels in comparison to most existing reference-free metrics and closely competes with zero-shot summary evaluators based on GPT-3.5.