What Language(s) Does Aya-23 Think In? How Multilinguality Affects Internal Language Representations
This study addresses the problem of understanding internal language processing in multilingual LLMs for AI researchers, providing insights into cross-lingual transfer.
The researchers analyzed how Aya-23-8B, a multilingual LLM, processes language internally compared to monolingual models, finding that it activates related language representations during translation and shows specific neuron patterns for code-mixed inputs.
Large language models (LLMs) excel at multilingual tasks, yet their internal language processing remains poorly understood. We analyze how Aya-23-8B, a decoder-only LLM trained on balanced multilingual data, handles code-mixed, cloze, and translation tasks compared to predominantly monolingual models like Llama 3 and Chinese-LLaMA-2. Using logit lens and neuron specialization analyses, we find: (1) Aya-23 activates typologically related language representations during translation, unlike English-centric models that rely on a single pivot language; (2) code-mixed neuron activation patterns vary with mixing rates and are shaped more by the base language than the mixed-in one; and (3) Aya-23's languagespecific neurons for code-mixed inputs concentrate in final layers, diverging from prior findings on decoder-only models. Neuron overlap analysis further shows that script similarity and typological relations impact processing across model types. These findings reveal how multilingual training shapes LLM internals and inform future cross-lingual transfer research.