Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression
For practitioners deploying large MoE models, this method enables more aggressive compression by exploiting fine-grained channel-level redundancy, outperforming prior expert-level pruning approaches.
The paper tackles the problem of high memory and inference costs in Mixture-of-Experts (MoE) models. It proposes a structural pruning method that reformulates prune-ratio allocation as a channel-score coverage maximization problem, achieving up to 5.27x memory reduction on Qwen3-30B-A3B while preserving accuracy under 50% or 25% pruning with 4-bit quantization.
Mixture-of-Experts (MoE) models scale compute efficiently, yet remain expensive to deploy due to their substantial memory footprint and inference overhead. Prior compression methods mainly operate at the expert level, either removing entire experts or ranking experts by coarse-grained importance scores. However, such expert-wise decisions are often too coarse to capture fine-grained redundancy, leading to misallocated pruning budgets and limited compression. To address this problem, we observe that information within MoE experts is highly concentrated in a small subset of channels, leaving substantial redundancy even in experts deemed important. Based on this observation, we propose a structural pruning framework tailored for MoE models. Our method reformulates prune-ratio allocation as a channel-score coverage maximization problem and solves it efficiently using an attribution-based approximation. Experiments on DeepSeek and Qwen MoE models show that our method preserves model accuracy under 50% or 25% structured pruning when combined with 4-bit quantization. On Qwen3-30B-A3B, our approach reduces memory footprint by 5.27$\times$ and consistently outperforms state-of-the-art baselines across diverse benchmarks.