To Point or Not to Point: Understanding How Abstractive Summarizers Paraphrase Text
This research addresses the problem of understanding model strategies in abstractive summarization for NLP researchers, revealing incremental insights into their limitations.
The study investigated how abstractive summarization models, specifically the pointer-generator model, use copy/generation strategies to paraphrase text, finding that they rely heavily on syntactic boundaries for extraction and exhibit limited abstractive abilities with factual inaccuracies.
Abstractive neural summarization models have seen great improvements in recent years, as shown by ROUGE scores of the generated summaries. But despite these improved metrics, there is limited understanding of the strategies different models employ, and how those strategies relate their understanding of language. To understand this better, we run several experiments to characterize how one popular abstractive model, the pointer-generator model of See et al. (2017), uses its explicit copy/generation switch to control its level of abstraction (generation) vs extraction (copying). On an extractive-biased dataset, the model utilizes syntactic boundaries to truncate sentences that are otherwise often copied verbatim. When we modify the copy/generation switch and force the model to generate, only simple paraphrasing abilities are revealed alongside factual inaccuracies and hallucinations. On an abstractive-biased dataset, the model copies infrequently but shows similarly limited abstractive abilities. In line with previous research, these results suggest that abstractive summarization models lack the semantic understanding necessary to generate paraphrases that are both abstractive and faithful to the source document.