Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering
This work addresses the need to understand internal mechanisms in MoE models for multilingual natural language processing, providing insights that could enhance model efficiency and performance across languages, though it is incremental as it builds on existing MoE architectures.
The paper tackled the problem of understanding how Mixture-of-Experts (MoE) models achieve multilingual performance by analyzing routing behavior and expert specialization, revealing structured patterns such as routing alignment with linguistic families and layerwise processing roles, and proposed a routing-guided steering method that improved multilingual performance, particularly for related language pairs.
Mixture-of-Experts (MoE) architectures have shown strong multilingual capabilities, yet the internal mechanisms underlying performance gains and cross-language differences remain insufficiently understood. In this work, we conduct a systematic analysis of MoE models, examining routing behavior and expert specialization across languages and network depth. Our analysis reveals that multilingual processing in MoE models is highly structured: routing aligns with linguistic families, expert utilization follows a clear layerwise pattern, and high-resource languages rely on shared experts while low-resource languages depend more on language-exclusive experts despite weaker performance. Layerwise interventions further show that early and late MoE layers support language-specific processing, whereas middle layers serve as language-agnostic capacity hubs. Building on these insights, we propose a routing-guided steering method that adaptively guides routing behavior in middle layers toward shared experts associated with dominant languages at inference time, leading to consistent multilingual performance improvements, particularly for linguistically related language pairs. Our code is available at https://github.com/conctsai/Multilingualism-in-Mixture-of-Experts-LLMs.