Language-Specific Latent Process Hinders Cross-Lingual Performance
This addresses the issue of unreliable cross-lingual transfer in LLMs for users requiring consistent multilingual AI applications, but it is incremental as it builds on existing analysis methods and focuses on improving small models.
The paper tackled the problem of inconsistent cross-lingual performance in large language models (LLMs) by analyzing representation similarity and latent processes, finding that models rely on language-specific representations rather than a shared semantic space, and demonstrated that steering small models towards shared representations improves multilingual reasoning performance and output consistency with English.
Large language models (LLMs) are demonstrably capable of cross-lingual transfer, but can produce inconsistent output when prompted with the same queries written in different languages. To understand how language models are able to generalize knowledge from one language to the others, we measure representation similarity between languages, and apply the logit lens to interpret the implicit steps taken by LLMs to solve multilingual multi-choice reasoning questions. Our analyses reveal LLMs predict inconsistently and are less accurate because they rely on representations that are dissimilar across languages, rather than working in a shared semantic space. While larger models are more multilingual, we show their hidden states are more likely to dissociate from the shared representation compared to smaller models, but are nevertheless more capable of retrieving knowledge embedded across different languages. Finally, we demonstrate that knowledge sharing in small models can be facilitated by steering their latent processing towards the shared semantic space. This improves the models' multilingual reasoning performance, as a result of more knowledge transfer from, and better output consistency with English.