Samoyeds: Accelerating MoE Models with Structured Sparsity Leveraging Sparse Tensor Cores
This addresses efficiency bottlenecks for deploying large MoE-based language models, offering a novel dual-side sparsity approach that improves performance and memory usage over prior methods.
The paper tackles the computational and memory challenges of large Mixture-of-Experts (MoE) models by introducing Samoyeds, an acceleration system that applies structured sparsity to both activations and parameters using Sparse Tensor Cores, achieving up to 1.99x kernel-level and 1.58x model-level speedups and increasing batch sizes by 4.41x on average.
The escalating size of Mixture-of-Experts (MoE) based Large Language Models (LLMs) presents significant computational and memory challenges, necessitating innovative solutions to enhance efficiency without compromising model accuracy. Structured sparsity emerges as a compelling strategy to address these challenges by leveraging the emerging sparse computing hardware. Prior works mainly focus on the sparsity in model parameters, neglecting the inherent sparse patterns in activations. This oversight can lead to additional computational costs associated with activations, potentially resulting in suboptimal performance. This paper presents Samoyeds, an innovative acceleration system for MoE LLMs utilizing Sparse Tensor Cores (SpTCs). Samoyeds is the first to apply sparsity simultaneously to both activations and model parameters. It introduces a bespoke sparse data format tailored for MoE computation and develops a specialized sparse-sparse matrix multiplication kernel. Furthermore, Samoyeds incorporates systematic optimizations specifically designed for the execution of dual-side structured sparse MoE LLMs on SpTCs, further enhancing system performance. Evaluations show that Samoyeds outperforms SOTA works by up to 1.99$\times$ at the kernel level and 1.58$\times$ at the model level. Moreover, it enhances memory efficiency, increasing maximum supported batch sizes by 4.41$\times$ on average. Additionally, Samoyeds surpasses existing SOTA structured sparse solutions in both model accuracy and hardware portability.