The New LLM Bottleneck: A Systems Perspective on Latent Attention and Mixture-of-Experts
This work addresses the systems design challenge for next-generation Transformers, shifting focus from accelerating memory-bound layers to creating balanced systems for large-scale models.
The paper argues that recent architectural shifts like Multi-head Latent Attention and Mixture-of-Experts reduce the need for specialized attention hardware by increasing arithmetic intensity, making these components more compute-bound and better suited for modern accelerators like GPUs.
Computational workloads composing traditional Transformer models are starkly bifurcated. Multi-Head Attention (MHA) is memory-bound, with low arithmetic intensity, while feedforward layers are compute-bound. This dichotomy has long motivated research into specialized hardware to mitigate the MHA bottleneck. This paper argues that recent architectural shifts, namely Multi-head Latent Attention (MLA) and Mixture-of-Experts (MoE), challenge the premise of specialized attention hardware. We make two key observations. First, the arithmetic intensity of MLA is over two orders of magnitude greater than that of MHA, shifting it close to a compute-bound regime well-suited for modern accelerators like GPUs. Second, by distributing MoE experts across a pool of accelerators, their arithmetic intensity can be tuned through batching to match that of the dense layers, creating a more balanced computational profile. These findings reveal a diminishing need for specialized attention hardware. The central challenge for next-generation Transformers is no longer accelerating a single memory-bound layer. Instead, the focus must shift to designing balanced systems with sufficient compute, memory capacity, memory bandwidth, and high-bandwidth interconnects to manage the diverse demands of large-scale models.