Attention Optimization for Abstractive Document Summarization
This work addresses the problem of generating high-quality summaries by optimizing attention for researchers and practitioners in NLP, but it is incremental as it builds on existing sequence-to-sequence models.
The paper tackled improving attention mechanisms in abstractive document summarization by augmenting vanilla attention with local and global variance losses, resulting in enhanced performance on the CNN/Daily Mail dataset.
Attention plays a key role in the improvement of sequence-to-sequence-based document summarization models. To obtain a powerful attention helping with reproducing the most salient information and avoiding repetitions, we augment the vanilla attention model from both local and global aspects. We propose an attention refinement unit paired with local variance loss to impose supervision on the attention model at each decoding step, and a global variance loss to optimize the attention distributions of all decoding steps from the global perspective. The performances on the CNN/Daily Mail dataset verify the effectiveness of our methods.