Language Modeling using LMUs: 10x Better Data Efficiency or Improved Scaling Compared to Transformers
This addresses the problem of data inefficiency and scaling limitations in language modeling for AI researchers and practitioners, offering a novel architecture that could reduce training costs.
The paper tackles the high data and computational demands of transformers in language modeling by proposing a Legendre Memory Unit (LMU)-based model with linear or near-linear dependencies, achieving the same accuracy as transformers with 10x fewer tokens and improving loss comparably to transformers over LSTMs.
Recent studies have demonstrated that the performance of transformers on the task of language modeling obeys a power-law relationship with model size over six orders of magnitude. While transformers exhibit impressive scaling, their performance hinges on processing large amounts of data, and their computational and memory requirements grow quadratically with sequence length. Motivated by these considerations, we construct a Legendre Memory Unit based model that introduces a general prior for sequence processing and exhibits an $O(n)$ and $O(n \ln n)$ (or better) dependency for memory and computation respectively. Over three orders of magnitude, we show that our new architecture attains the same accuracy as transformers with 10x fewer tokens. We also show that for the same amount of training our model improves the loss over transformers about as much as transformers improve over LSTMs. Additionally, we demonstrate that adding global self-attention complements our architecture and the augmented model improves performance even further.