SEAL: Segment-wise Extractive-Abstractive Long-form Text Summarization
It addresses the challenge of summarizing very long documents for applications like research or news, offering improved performance and interpretability, though it builds incrementally on existing Transformer methods.
The paper tackles long-form abstractive text summarization with input sequences up to 100,000 tokens by proposing SEAL, a Transformer-based model that dynamically extracts input snippets for each output segment, achieving state-of-the-art results on existing tasks and outperforming baselines on a new dataset with longer text.
Most prior work in the sequence-to-sequence paradigm focused on datasets with input sequence lengths in the hundreds of tokens due to the computational constraints of common RNN and Transformer architectures. In this paper, we study long-form abstractive text summarization, a sequence-to-sequence setting with input sequence lengths up to 100,000 tokens and output sequence lengths up to 768 tokens. We propose SEAL, a Transformer-based model, featuring a new encoder-decoder attention that dynamically extracts/selects input snippets to sparsely attend to for each output segment. Using only the original documents and summaries, we derive proxy labels that provide weak supervision for extractive layers simultaneously with regular supervision from abstractive summaries. The SEAL model achieves state-of-the-art results on existing long-form summarization tasks, and outperforms strong baseline models on a new dataset/task we introduce, Search2Wiki, with much longer input text. Since content selection is explicit in the SEAL model, a desirable side effect is that the selection can be inspected for enhanced interpretability.