Language-specific Neurons Do Not Facilitate Cross-Lingual Transfer
This work addresses the challenge of performance degradation in low-resource languages for multilingual LLMs, though it is incremental as it tests existing techniques without proposing new solutions.
The study investigated whether identifying language-specific neurons in multilingual LLMs could improve cross-lingual task performance for low-resource languages, but found that interventions like neuron-specific fine-tuning did not yield improvements on tasks such as XNLI and XQuAD.
Multilingual large language models (LLMs) aim towards robust natural language understanding across diverse languages, yet their performance significantly degrades on low-resource languages. This work explores whether existing techniques to identify language-specific neurons can be leveraged to enhance cross-lingual task performance of lowresource languages. We conduct detailed experiments covering existing language-specific neuron identification techniques (such as Language Activation Probability Entropy and activation probability-based thresholding) and neuron-specific LoRA fine-tuning with models like Llama 3.1 and Mistral Nemo. We find that such neuron-specific interventions are insufficient to yield cross-lingual improvements on downstream tasks (XNLI, XQuAD) in lowresource languages. This study highlights the challenges in achieving cross-lingual generalization and provides critical insights for multilingual LLMs.