UMoE: Unifying Attention and FFN with Shared Experts
This work addresses a bottleneck in scaling Transformer models for AI researchers and practitioners, offering an incremental improvement over existing MoE methods.
The paper tackled the suboptimal performance of attention-based Mixture of Experts (MoE) layers in Transformers by introducing UMoE, which unifies MoE designs in attention and feed-forward network (FFN) layers through a novel reformulation of attention as FFN-like structures, achieving superior performance with efficient parameter sharing.
Sparse Mixture of Experts (MoE) architectures have emerged as a promising approach for scaling Transformer models. While initial works primarily incorporated MoE into feed-forward network (FFN) layers, recent studies have explored extending the MoE paradigm to attention layers to enhance model performance. However, existing attention-based MoE layers require specialized implementations and demonstrate suboptimal performance compared to their FFN-based counterparts. In this paper, we aim to unify MoE designs in attention and FFN layers by introducing a novel reformulation of the attention mechanism, that reveals an underlying FFN-like structure within attention modules. Our proposed architecture, UMoE, achieves superior performance through attention-based MoE layers while enabling efficient parameter sharing between FFN and attention components.