Extended Mind Transformers
This addresses the problem of memory limitations in language models for AI researchers and practitioners, though it is incremental as it builds on existing Memorizing Transformers.
The paper tackles the bottleneck of long inputs in pre-trained language models by improving Memorizing Transformers, fixing issues like the need for fine-tuning through better positional encoding updates and using the model's own key/query system for memory retrieval. It shows that Extended Mind Transformers outperform state-of-the-art methods by 6% on average on a new benchmark.
Pre-trained language models demonstrate general intelligence and common sense, but long inputs quickly become a bottleneck for memorizing information at inference time. We resurface a simple method, Memorizing Transformers (Wu et al., 2022), that gives the model access to a bank of pre-computed memories. We show that it is possible to fix many of the shortcomings of the original method, such as the need for fine-tuning, by critically assessing how positional encodings should be updated for the keys and values retrieved. This intuitive method uses the model's own key/query system to select and attend to the most relevant memories at each generation step, rather than using external embeddings. We demonstrate the importance of external information being retrieved in a majority of decoder layers, contrary to previous work. We open source a new counterfactual long-range retrieval benchmark, and show that Extended Mind Transformers outperform today's state of the art by 6% on average.