Learning to Summarize Long Texts with Memory Compression and Transfer
This work addresses the challenge of summarizing long documents efficiently for natural language processing applications, offering a more parameter-efficient approach compared to existing methods.
The authors tackled the problem of abstractive document summarization for long texts by introducing Mem2Mem, a memory-to-memory mechanism that compresses input articles into compact sentence representations without extraction labels, achieving competitive results with state-of-the-art transformer methods while using 16 times fewer parameters.
We introduce Mem2Mem, a memory-to-memory mechanism for hierarchical recurrent neural network based encoder decoder architectures and we explore its use for abstractive document summarization. Mem2Mem transfers "memories" via readable/writable external memory modules that augment both the encoder and decoder. Our memory regularization compresses an encoded input article into a more compact set of sentence representations. Most importantly, the memory compression step performs implicit extraction without labels, sidestepping issues with suboptimal ground-truth data and exposure bias of hybrid extractive-abstractive summarization techniques. By allowing the decoder to read/write over the encoded input memory, the model learns to read salient information about the input article while keeping track of what has been generated. Our Mem2Mem approach yields results that are competitive with state of the art transformer based summarization methods, but with 16 times fewer parameters