Associative Recurrent Memory Transformer
This addresses the problem of efficient long-sequence processing for AI/ML applications, representing a strong incremental improvement over prior methods.
The paper tackles the challenge of processing very long sequences with constant time per step by introducing the Associative Recurrent Memory Transformer (ARMT), which combines transformer self-attention with segment-level recurrence. It achieves 79.9% accuracy on the BABILong benchmark for single-fact questions over 50 million tokens, outperforming existing alternatives.
This paper addresses the challenge of creating a neural architecture for very long sequences that requires constant time for processing new information at each time step. Our approach, Associative Recurrent Memory Transformer (ARMT), is based on transformer self-attention for local context and segment-level recurrence for storage of task specific information distributed over a long context. We demonstrate that ARMT outperfors existing alternatives in associative retrieval tasks and sets a new performance record in the recent BABILong multi-task long-context benchmark by answering single-fact questions over 50 million tokens with an accuracy of 79.9%. The source code for training and evaluation is available on github.