Learning Cluster Patterns for Abstractive Summarization
This work addresses the challenge of enhancing summary quality in natural language processing, but it is incremental as it builds on existing pre-trained models like BART.
The paper tackles the problem of improving abstractive summarization by learning distinct cluster patterns for salient and non-salient context vectors, resulting in up to 4% improvement in ROUGE and 0.3% in BERTScore on average over the BART model on CNN/DailyMail and XSUM datasets.
Nowadays, pre-trained sequence-to-sequence models such as BERTSUM and BART have shown state-of-the-art results in abstractive summarization. In these models, during fine-tuning, the encoder transforms sentences to context vectors in the latent space and the decoder learns the summary generation task based on the context vectors. In our approach, we consider two clusters of salient and non-salient context vectors, using which the decoder can attend more to salient context vectors for summary generation. For this, we propose a novel clustering transformer layer between the encoder and the decoder, which first generates two clusters of salient and non-salient vectors, and then normalizes and shrinks the clusters to make them apart in the latent space. Our experimental result shows that the proposed model outperforms the existing BART model by learning these distinct cluster patterns, improving up to 4% in ROUGE and 0.3% in BERTScore on average in CNN/DailyMail and XSUM data sets.