MH-MoE: Multi-Head Mixture-of-Experts
This work addresses efficiency and performance challenges in large language models for AI researchers, though it appears incremental as it builds on existing MoE frameworks.
The paper tackles the problem of improving Mixture-of-Experts (MoE) models by introducing MH-MoE, which uses a multi-head mechanism to enhance performance while maintaining computational and parameter efficiency, resulting in quality improvements over existing MoE models in language modeling experiments.
Multi-Head Mixture-of-Experts (MH-MoE) demonstrates superior performance by using the multi-head mechanism to collectively attend to information from various representation spaces within different experts. In this paper, we present a novel implementation of MH-MoE that maintains both FLOPs and parameter parity with sparse Mixture of Experts models. Experimental results on language models show that the new implementation yields quality improvements over both vanilla MoE and fine-grained MoE models. Additionally, our experiments demonstrate that MH-MoE is compatible with 1-bit Large Language Models (LLMs) such as BitNet.