MergeMoE: Efficient Compression of MoE Models via Expert Output Merging
This addresses memory efficiency for large language models using MoE, but it is incremental as it builds on existing expert merging techniques.
The paper tackles the problem of compressing Mixture-of-Experts (MoE) models to reduce memory overhead by proposing MergeMoE, a method that merges expert outputs using mathematical optimization, and shows it consistently outperforms baselines at the same compression ratios.
The Mixture-of-Experts (MoE) technique has proven to be a promising solution to efficiently scale the model size, which has been widely applied in recent LLM advancements. However, the substantial memory overhead of MoE models has made their compression an important research direction. In this work, we provide a theoretical analysis of expert merging, a recently proposed technique for compressing MoE models. Rather than interpreting expert merging from the conventional perspective of parameter aggregation, we approach it from the perspective of merging experts' outputs. Our key insight is that the merging process can be interpreted as inserting additional matrices into the forward computation, which naturally leads to an optimization formulation. Building on this analysis, we introduce MergeMoE, a method that leverages mathematical optimization to construct the compression matrices. We evaluate MergeMoE on multiple MoE models and show that our algorithm consistently outperforms the baselines with the same compression ratios.