Point-less: More Abstractive Summarization with Pointer-Generator Networks
This work addresses the issue of low abstractiveness in summarization models for NLP researchers, but it is incremental as it builds on existing Pointer-Generator architectures.
The authors tackled the problem of Pointer-Generator networks producing overly extractive summaries, where over 30% of sentences were copied from source text, by proposing new techniques like multihead attention and loss functions, resulting in significantly higher novel N-grams and sentences but a slightly lower ROUGE score.
The Pointer-Generator architecture has shown to be a big improvement for abstractive summarization seq2seq models. However, the summaries produced by this model are largely extractive as over 30% of the generated sentences are copied from the source text. This work proposes a multihead attention mechanism, pointer dropout, and two new loss functions to promote more abstractive summaries while maintaining similar ROUGE scores. Both the multihead attention and dropout do not improve N-gram novelty, however, the dropout acts as a regularizer which improves the ROUGE score. The new loss function achieves significantly higher novel N-grams and sentences, at the cost of a slightly lower ROUGE score.