On Extractive and Abstractive Neural Document Summarization with Transformer Language Models
This work addresses the challenge of summarizing lengthy documents for applications like news or research, though it appears incremental as it builds on existing transformer-based approaches.
The paper tackles the problem of generating abstractive summaries for long documents by introducing a simple extractive step before using a transformer language model, which significantly improves summarization results and produces more abstractive summaries with higher ROUGE scores compared to prior methods.
We present a method to produce abstractive summaries of long documents that exceed several thousand words via neural abstractive summarization. We perform a simple extractive step before generating a summary, which is then used to condition the transformer language model on relevant information before being tasked with generating a summary. We show that this extractive step significantly improves summarization results. We also show that this approach produces more abstractive summaries compared to prior work that employs a copy mechanism while still achieving higher rouge scores. Note: The abstract above was not written by the authors, it was generated by one of the models presented in this paper.