Compressed Convolutional Attention: Efficient Attention in a Compressed Latent Space
This addresses efficiency bottlenecks in training and serving large transformers for users in AI and NLP, offering a novel hybrid approach that is incremental but with strong practical gains.
The paper tackles the high computational and memory costs of Multi-headed Attention in long-context transformers by introducing Compressed Convolutional Attention (CCA), which performs attention in a compressed latent space, reducing parameters, KV-cache, and FLOPs. Experiments show that CCGQA outperforms prior methods like GQA and MLA, achieving up to 8x KV-cache compression without performance loss and reducing prefill latency by about 1.7x on H100 GPUs.
Multi-headed Attention's (MHA) quadratic compute and linearly growing KV-cache make long-context transformers expensive to train and serve. Prior works such as Grouped Query Attention (GQA) and Multi-Latent Attention (MLA) shrink the cache, speeding decode, but leave compute, which determines prefill and training speed, largely unchanged. We introduce Compressed Convolutional Attention (CCA), a novel attention method which down-projects queries, keys, and values and performs the entire attention operation inside the shared latent space. This simple design dramatically cuts parameters, KV-cache, and FLOPs all at once by the desired compression factor. Because CCA is orthogonal to head-sharing, we combine the two to form Compressed Convolutional Grouped Query Attention (CCGQA), which further tightens the compute-bandwidth Pareto frontier so that users can tune compression toward either FLOP or memory limits without sacrificing quality. Experiments show that CCGQA consistently outperforms both GQA and MLA at equal KV-cache compression on dense and MoE models. Additionally, we show that CCGQA outperforms all other attention methods on MoE models with half the KV-cache of GQA and MLA, achieving an 8x KV-cache compression with no drop in performance compared to standard MHA. CCA and CCGQA also dramatically reduce the FLOP cost of attention which leads to substantially faster training and prefill than existing methods. On H100 GPUs, our fused CCA/CCGQA kernel reduces prefill latency by about 1.7x at a sequence length of 16k relative to MHA, and accelerates backward by about 1.3x.