HiStruct+: Improving Extractive Text Summarization with Hierarchical Structure Information
This work addresses the limitation of linear text processing in summarization for domains with clear hierarchical structures, such as scientific articles, though it is incremental in building on existing Transformer methods.
The authors tackled the problem of extractive text summarization by incorporating hierarchical structure information into Transformer models, achieving state-of-the-art ROUGE scores on PubMed and arXiv datasets with substantial improvements.
Transformer-based language models usually treat texts as linear sequences. However, most texts also have an inherent hierarchical structure, i.e., parts of a text can be identified using their position in this hierarchy. In addition, section titles usually indicate the common topic of their respective sentences. We propose a novel approach to formulate, extract, encode and inject hierarchical structure information explicitly into an extractive summarization model based on a pre-trained, encoder-only Transformer language model (HiStruct+ model), which improves SOTA ROUGEs for extractive summarization on PubMed and arXiv substantially. Using various experimental settings on three datasets (i.e., CNN/DailyMail, PubMed and arXiv), our HiStruct+ model outperforms a strong baseline collectively, which differs from our model only in that the hierarchical structure information is not injected. It is also observed that the more conspicuous hierarchical structure the dataset has, the larger improvements our method gains. The ablation study demonstrates that the hierarchical position information is the main contributor to our model's SOTA performance.