Compact Recurrent Transformer with Persistent Memory
This addresses the computational bottleneck for long sequences in Transformers, particularly for edge computing applications, though it is incremental as it builds on existing segment-based and memory mechanisms.
The paper tackles the inefficiency of Transformers on long sequences by proposing a Compact Recurrent Transformer that combines shallow Transformers with RNNs to manage a persistent memory vector, achieving comparable or superior results on language tasks with half or quarter segment sizes and reduced FLOPs, and state-of-the-art performance on a video dataset.
The Transformer architecture has shown significant success in many language processing and visual tasks. However, the method faces challenges in efficiently scaling to long sequences because the self-attention computation is quadratic with respect to the input length. To overcome this limitation, several approaches scale to longer sequences by breaking long sequences into a series of segments, restricting self-attention to local dependencies between tokens within each segment and using a memory mechanism to manage information flow between segments. However, these approached generally introduce additional compute overhead that restricts them from being used for applications where limited compute memory and power are of great concern (such as edge computing). We propose a novel and efficient Compact Recurrent Transformer (CRT), which combines shallow Transformer models that process short local segments with recurrent neural networks to compress and manage a single persistent memory vector that summarizes long-range global information between segments. We evaluate CRT on WordPTB and WikiText-103 for next-token-prediction tasks, as well as on the Toyota Smarthome video dataset for classification. CRT achieves comparable or superior prediction results to full-length Transformers in the language datasets while using significantly shorter segments (half or quarter size) and substantially reduced FLOPs. Our approach also demonstrates state-of-the-art performance on the Toyota Smarthome video dataset.