NeuronMoE: Neuron-Guided Mixture-of-Experts for Efficient Multilingual LLM Extension
This work offers a more efficient way to extend large language models to low-resource languages, which is important for improving global accessibility of LLMs, especially for communities with limited digital resources.
This paper addresses the high cost of extending large language models to low-resource languages by proposing NeuronMoE, a method that uses neuron-level analysis to guide the allocation of language-specific experts in Mixture-of-Experts architectures. Applied to Llama-3.2-3B for Greek, Turkish, and Hungarian, NeuronMoE achieved approximately 40% average parameter reduction while matching the performance of a LayerMoE baseline.
Extending large language models to low-resource languages is essential for global accessibility, but training separate models per language is prohibitively expensive. Mixture-of-Experts (MoE) architectures address this by adding sparse language-specific parameters, but determining how many experts each layer needs remains an open question. Current approaches allocate experts based on layer-level similarity, yet language processing exhibits fine-grained specialization at individual neurons. We propose $\textbf{NeuronMoE}$, a method that analyzes language-specific neurons across all transformer components to guide expert allocation per layer based on empirically measured cross-lingual neuron diversity. Applied to Llama-3.2-3B for low-resource languages (Greek, Turkish, and Hungarian), this approach achieves approximately 40% average parameter reduction while matching the performance of the LayerMoE baseline. We find that low-resource language experts independently develop neuron specialization patterns mirroring the high-resource language, which are concentrated in early and late layers. This reveals potential universal architectural principles in how multilingual models organize linguistic knowledge.