GRETEL: Graph Contrastive Topic Enhanced Language Model for Long Document Extractive Summarization
This work addresses the challenge of generating more salient and less redundant summaries for long documents, which is important for applications like information retrieval and content analysis, though it appears incremental as it builds on existing neural topic models and pre-trained language models.
The authors tackled the problem of capturing and integrating global semantic information for long document extractive summarization by proposing GRETEL, which incorporates a graph contrastive topic model with a pre-trained language model, and it outperformed state-of-the-art methods on general and biomedical datasets.
Recently, neural topic models (NTMs) have been incorporated into pre-trained language models (PLMs), to capture the global semantic information for text summarization. However, in these methods, there remain limitations in the way they capture and integrate the global semantic information. In this paper, we propose a novel model, the graph contrastive topic enhanced language model (GRETEL), that incorporates the graph contrastive topic model with the pre-trained language model, to fully leverage both the global and local contextual semantics for long document extractive summarization. To better capture and incorporate the global semantic information into PLMs, the graph contrastive topic model integrates the hierarchical transformer encoder and the graph contrastive learning to fuse the semantic information from the global document context and the gold summary. To this end, GRETEL encourages the model to efficiently extract salient sentences that are topically related to the gold summary, rather than redundant sentences that cover sub-optimal topics. Experimental results on both general domain and biomedical datasets demonstrate that our proposed method outperforms SOTA methods.