MoE Lens -- An Expert Is All You Need
This research provides insights into expert specialization in MoE models, which could lead to inference optimization through targeted expert pruning for practitioners working with large language models.
This paper analyzes Mixture of Experts (MoE) models, specifically DeepSeekMoE, and finds that despite having 64 experts with 6 active per layer, only a few specialized experts are predominantly used. The top-weighted expert's output closely approximates the full ensemble prediction, with some layers showing cosine similarity as high as 0.95 and only a 5% increase in perplexity when using a single expert.
Mixture of Experts (MoE) models enable parameter-efficient scaling through sparse expert activations, yet optimizing their inference and memory costs remains challenging due to limited understanding of their specialization behavior. We present a systematic analysis of expert specialization in MoEs through two complementary approaches: domain-specific routing patterns and an early decoding framework that tracks expert contributions to output representations. Our analysis of the DeepSeekMoE model reveals that despite having 64 routed experts with 6 active for each layer's computation, the model predominantly relies on a few specialized experts, with the top-weighted expert's output closely approximating the full ensemble prediction. We quantitatively validate these findings through a systematic analysis of the token routing distribution, demonstrating that very few experts handle over 50\% of routing decisions across different specialized domains. Hidden state similarity between single and ensemble experts for every layer is extremely high, with some layers having cosine similarity as high as 0.95 and perplexity increasing by only 5\% when using a single expert across all three domains. Our results indicate that Mixture of Experts models exhibit concentrated expertise highlighting potential opportunities for inference optimization through targeted expert pruning while maintaining model performance and opening avenues towards studying localization of learned knowledge in these models.