Improving the Downstream Performance of Mixture-of-Experts Transformers via Weak Vanilla Transformers
This addresses a practical limitation for users of MoE models in NLP by improving their downstream task applicability, though it is incremental as it builds on existing distillation and MoE techniques.
The paper tackles the problem of Mixture-of-Experts (MoE) Transformers underperforming vanilla Transformers in downstream tasks by proposing transfer capability distillation, where vanilla models guide MoE models to improve transfer capability, resulting in significant downstream performance gains as shown in experiments on BERT.
Recently, Mixture of Experts (MoE) Transformers have garnered increasing attention due to their advantages in model capacity and computational efficiency. However, studies have indicated that MoE Transformers underperform vanilla Transformers in many downstream tasks, significantly diminishing the practical value of MoE models. To explain this issue, we propose that the pre-training performance and transfer capability of a model are joint determinants of its downstream task performance. MoE models, in comparison to vanilla models, have poorer transfer capability, leading to their subpar performance in downstream tasks. To address this issue, we introduce the concept of transfer capability distillation, positing that although vanilla models have weaker performance, they are effective teachers of transfer capability. The MoE models guided by vanilla models can achieve both strong pre-training performance and transfer capability, ultimately enhancing their performance in downstream tasks. We design a specific distillation method and conduct experiments on the BERT architecture. Experimental results show a significant improvement in downstream performance of MoE models, and many further evidences also strongly support the concept of transfer capability distillation. Finally, we attempt to interpret transfer capability distillation and provide some insights from the perspective of model feature.