When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute
This addresses the problem of reducing compute time and cost for training language models, offering a promising direction for acceleration, though it appears incremental as it builds on existing recurrence and attention methods.
The authors tackled the high computational cost of training large language models by introducing SRU++, an architecture combining fast recurrence and attention, achieving better performance on language modeling tasks like Enwik8 with 3x-10x less training cost, such as a state-of-the-art result in 1.6 days on an 8-GPU machine.
Large language models have become increasingly difficult to train because of the growing computation time and cost. In this work, we present SRU++, a highly-efficient architecture that combines fast recurrence and attention for sequence modeling. SRU++ exhibits strong modeling capacity and training efficiency. On standard language modeling tasks such as Enwik8, Wiki-103 and Billion Word datasets, our model obtains better bits-per-character and perplexity while using 3x-10x less training cost compared to top-performing Transformer models. For instance, our model achieves a state-of-the-art result on the Enwik8 dataset using 1.6 days of training on an 8-GPU machine. We further demonstrate that SRU++ requires minimal attention for near state-of-the-art performance. Our results suggest jointly leveraging fast recurrence with little attention as a promising direction for accelerating model training and inference.