xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity
This work addresses the computational efficiency problem for large language model developers by showing that xLSTM offers a scalable alternative to Transformers, though it is incremental as it builds on existing xLSTM research.
The paper investigates the scaling behavior of xLSTM compared to Transformers, finding that xLSTM scales favorably with linear time-complexity and its advantage increases with longer training and inference contexts, achieving competitive performance in the billion-parameter regime.
Scaling laws play a central role in the success of Large Language Models (LLMs), enabling the prediction of model performance relative to compute budgets prior to training. While Transformers have been the dominant architecture, recent alternatives such as xLSTM offer linear complexity with respect to context length while remaining competitive in the billion-parameter regime. We conduct a comparative investigation on the scaling behavior of Transformers and xLSTM along the following lines, providing insights to guide future model design and deployment. First, we study the scaling behavior for xLSTM in compute-optimal and over-training regimes using both IsoFLOP and parametric fit approaches on a wide range of model sizes (80M-7B) and number of training tokens (2B-2T). Second, we examine the dependence of optimal model sizes on context length, a pivotal aspect that was largely ignored in previous work. Finally, we analyze inference-time scaling characteristics. Our findings reveal that in typical LLM training and inference scenarios, xLSTM scales favorably compared to Transformers. Importantly, xLSTM's advantage widens as training and inference contexts grow.