Task-Specific Expert Pruning for Sparse Mixture-of-Experts
This addresses the problem of high computational and communication costs for deploying large MoE models in resource-limited environments, representing an incremental improvement.
The paper tackles the deployment inefficiency of sparse Mixture-of-Experts (MoE) models by proposing a method to prune non-professional experts for downstream tasks, resulting in a single-expert dense model that preserves 99.3% of MoE benefits and achieves 2x inference speed with no communication cost.
The sparse Mixture-of-Experts (MoE) model is powerful for large-scale pre-training and has achieved promising results due to its model capacity. However, with trillions of parameters, MoE is hard to be deployed on cloud or mobile environment. The inference of MoE requires expert parallelism, which is not hardware-friendly and communication expensive. Especially for resource-limited downstream tasks, such sparse structure has to sacrifice a lot of computing efficiency for limited performance gains. In this work, we observe most experts contribute scarcely little to the MoE fine-tuning and inference. We further propose a general method to progressively drop the non-professional experts for the target downstream task, which preserves the benefits of MoE while reducing the MoE model into one single-expert dense model. Our experiments reveal that the fine-tuned single-expert model could preserve 99.3% benefits from MoE across six different types of tasks while enjoying 2x inference speed with free communication cost.