HETFORMER: Heterogeneous Transformer with Sparse Attention for Long-Text Extractive Summarization
This addresses the challenge of long-text summarization for NLP practitioners, offering a more efficient method, though it appears incremental as it builds on existing Transformer and sparse attention techniques.
The paper tackled the problem of inefficient computations and cumbersome procedures in long-text extractive summarization by proposing HETFORMER, a Transformer-based model with multi-granularity sparse attentions, achieving state-of-the-art performance in Rouge F1 with less memory and fewer parameters.
To capture the semantic graph structure from raw text, most existing summarization approaches are built on GNNs with a pre-trained model. However, these methods suffer from cumbersome procedures and inefficient computations for long-text documents. To mitigate these issues, this paper proposes HETFORMER, a Transformer-based pre-trained model with multi-granularity sparse attentions for long-text extractive summarization. Specifically, we model different types of semantic nodes in raw text as a potential heterogeneous graph and directly learn heterogeneous relationships (edges) among nodes by Transformer. Extensive experiments on both single- and multi-document summarization tasks show that HETFORMER achieves state-of-the-art performance in Rouge F1 while using less memory and fewer parameters.