Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddings
This research addresses the problem of understanding cross-lingual interpretability in LLMs for AI and linguistics, offering insights into model behavior without explicit cross-lingual training.
The study investigated how multilingual large language models (LLMs) represent relationships between languages through token embeddings, finding that XLM-RoBERTa embeddings encode language with 99.2% accuracy in linear separation, while mT5 embeddings capture cross-lingual semantic similarity with nearest neighbors averaging 7.61 writing systems.
Cross-lingual transfer learning is an important property of multilingual large language models (LLMs). But how do LLMs represent relationships between languages? Every language model has an input layer that maps tokens to vectors. This ubiquitous layer of language models is often overlooked. We find that similarities between these input embeddings are highly interpretable and that the geometry of these embeddings differs between model families. In one case (XLM-RoBERTa), embeddings encode language: tokens in different writing systems can be linearly separated with an average of 99.2% accuracy. Another family (mT5) represents cross-lingual semantic similarity: the 50 nearest neighbors for any token represent an average of 7.61 writing systems, and are frequently translations. This result is surprising given that there is no explicit parallel cross-lingual training corpora and no explicit incentive for translations in pre-training objectives. Our research opens the door for investigations in 1) The effect of pre-training and model architectures on representations of languages and 2) The applications of cross-lingual representations embedded in language models.