How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle
This work provides a principled framework for selecting pruning criteria in MoE models, addressing a gap for practitioners needing to reduce memory usage while maintaining performance.
The paper addresses the lack of principled criteria for one-shot expert pruning in MoE language models, proposing a unified formulation that yields a selection principle and two new task-agnostic criteria (MAN and MSAN). These criteria achieve top-two average ranks across four MoE models and 16 benchmarks, improving average performance by up to 8.8 points over the strongest baseline.
Mixture-of-Experts (MoE) language models reduce per-token computation through sparse expert activation, yet deployment still requires storing the full expert pool, making one-shot expert pruning a practical approach for reducing memory usage. Although effective, existing criteria are largely heuristic, and no single criterion is universally optimal. Thus, establishing a principle for selecting pruning criteria suited to different deployment objectives remains an important yet largely underexplored problem in one-shot expert pruning. To this end, we introduce a unified formulation for one-shot MoE expert pruning organized around three factors: routing frequency, gate weighting, and activation strength. The formulation yields a criteria selection principle: task-agnostic pruning should favor routed-token-averaged, gate-free activation-based criteria, whereas task-specific pruning can benefit from retaining routing-frequency and gate-weight information. Beyond this principle, the formulation also provides a systematic view of existing heuristic criteria and gives rise to two new task-agnostic criteria, Mean Activation Norm (MAN) and Mean Squared Activation Norm (MSAN). Across four representative MoE models and 16 diverse benchmarks, MAN and MSAN are consistently strong in the task-agnostic setting, obtain the top-two average ranks, and improve average performance by up to 8.8 points over the strongest baseline.