Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization
This addresses efficiency problems for large language model deployment by improving dynamic resource allocation, though it builds incrementally on existing KV optimization methods.
The paper tackles the scalability challenge in transformer models caused by inefficient memory allocation for key-value caches by proposing mixSGA, a mixture-of-experts approach that dynamically optimizes token-wise computation and memory allocation without discarding tokens. It achieves higher ROUGE-L scores and lower perplexity under the same KV budgets compared to static baselines in evaluations across multiple model families.
Transformer models face scalability challenges in causal language modeling (CLM) due to inefficient memory allocation for growing key-value (KV) caches, which strains compute and storage resources. Existing methods like Grouped Query Attention (GQA) and token-level KV optimization improve efficiency but rely on rigid resource allocation, often discarding "low-priority" tokens or statically grouping them, failing to address the dynamic spectrum of token importance. We propose mixSGA, a novel mixture-of-expert (MoE) approach that dynamically optimizes token-wise computation and memory allocation. Unlike prior approaches, mixSGA retains all tokens while adaptively routing them to specialized experts with varying KV group sizes, balancing granularity and efficiency. Our key novelties include: (1) a token-wise expert-choice routing mechanism guided by learned importance scores, enabling proportional resource allocation without token discard; (2) weight-sharing across grouped attention projections to minimize parameter overhead; and (3) an auxiliary loss to ensure one-hot routing decisions for training-inference consistency in CLMs. Extensive evaluations across Llama3, TinyLlama, OPT, and Gemma2 model families show mixSGA's superiority over static baselines. On instruction-following and continued pretraining tasks, mixSGA achieves higher ROUGE-L and lower perplexity under the same KV budgets.