How does Alignment Enhance LLMs' Multilingual Capabilities? A Language Neurons Perspective
This work provides insights into multilingual alignment mechanisms for LLM researchers, but it is incremental as it builds on existing neuron analysis methods.
The paper tackled the problem of understanding how alignment enhances multilingual capabilities in large language models (LLMs) by analyzing language-specific and language-agnostic neurons, and it found that a finer-grained neuron identification algorithm reveals distributional characteristics that divide multilingual inference into four parts, with empirical results provided.
Multilingual Alignment is an effective and representative paradigm to enhance LLMs' multilingual capabilities, which transfers the capabilities from the high-resource languages to the low-resource languages. Meanwhile, some researches on language-specific neurons reveal that there are language-specific neurons that are selectively activated in LLMs when processing different languages. This provides a new perspective to analyze and understand LLMs' mechanisms more specifically in multilingual scenarios. In this work, we propose a new finer-grained neuron identification algorithm, which detects language neurons~(including language-specific neurons and language-related neurons) and language-agnostic neurons. Furthermore, based on the distributional characteristics of different types of neurons, we divide the LLMs' internal process for multilingual inference into four parts: (1) multilingual understanding, (2) shared semantic space reasoning, (3) multilingual output space transformation, and (4) vocabulary space outputting. Additionally, we systematically analyze the models before and after alignment with a focus on different types of neurons. We also analyze the phenomenon of ''Spontaneous Multilingual Alignment''. Overall, our work conducts a comprehensive investigation based on different types of neurons, providing empirical results and valuable insights for better understanding multilingual alignment and multilingual capabilities of LLMs.