Memformer: A Memory-Augmented Transformer for Sequence Modeling
This addresses computational bottlenecks for researchers and practitioners working with long sequences in NLP and related fields, though it's an incremental improvement on existing Transformer architectures.
The paper tackles the efficiency problem of Transformers needing to store all history token-level representations by introducing Memformer, a memory-augmented model with linear time and constant memory complexity, achieving comparable performance with 8.1x less memory and 3.2x faster inference.
Transformers have reached remarkable success in sequence modeling. However, these models have efficiency issues as they need to store all the history token-level representations as memory. We present Memformer, an efficient neural network for sequence modeling, that utilizes an external dynamic memory to encode and retrieve past information. Our model achieves linear time complexity and constant memory space complexity when processing long sequences. We also propose a new optimization scheme, memory replay back-propagation (MRBP), which promotes long-range back-propagation through time with a significantly reduced memory requirement. Experimental results show that Memformer has achieved comparable performance compared to the baselines by using 8.1x less memory space and 3.2x faster on inference. Analysis of the attention pattern shows that our external memory slots can encode and retain important information through timesteps.