MC-SMoE

“However, this approach inherently diminishes the model's representational diversity, and identifying an optimal merging strategy is non-trivial. Furthermore, while MC-SMoE employs progressive low-rank decomposition during retraining for further expert compression, it introduces substantial training overhead.”

“When applied to MoE models with low-similarity experts, these methods generally fail due to significant parameter conflicts during the merging process.”

“While this reduces computational costs, it sacrifices the token-level routing flexibility that makes MoE models powerful.”

“existing expert pruning methods such as MC-SMoE li2024merge and RS he2024demystifying remove experts from MoE models primarily based on the expert access frequency. However, as shown in Figure intro (b), this feature alone fails to fully capture the expert redundancy”

“This idealized assumption often limits performance.”

“Nevertheless, in task-agnostic settings without retraining, relying on frequency information for clustering proves ineffective in Table~{tab:qwen and Table~tab:mixtral}. This approach faces three main issues. First, frequency varies across tasks, as shown in Appendix sec:freq-analysis, making it an unreliable indicator for deciding how many experts to retain in each layer. Second, high-frequency experts within the same layer are rarely merged, overlooking their functional similarities in the feature space. Moreover, grouping based on router information can be problematic, as it depends on dataset-dependent statistics.”

“More recently, MC-SMoE~li2024mergecompressdemystifyefficient dynamically merges experts during inference time, though it is limited to specific tasks.”

“M-SMoE demonstrates the potential of clustering and merging experts to reduce model size, but its merging algorithm is heuristic in nature and lacks theoretical support.”