High-Dimensional Interlingual Representations of Large Language Models
This addresses the challenge of scalable multilingual learning for AI systems by clarifying and enhancing interlingual representations, though it is incremental as it builds on existing multilingual model research.
The paper tackles the problem of inconsistent cross-lingual alignments in multilingual large language models by proposing an interlingual representation framework and a metric called Interlingual Local Overlap (ILO) to quantify alignment. It finds that single-language fine-tuning disrupts alignment in early layers, but freezing these layers preserves alignment and improves cross-lingual generalization.
Large language models (LLMs) trained on massive multilingual datasets hint at the formation of interlingual constructs--a shared subspace in the representation space. However, evidence regarding this phenomenon is mixed, leaving it unclear whether these models truly develop unified interlingual representations, or present a partially aligned constructs. We explore 31 diverse languages varying on their resource-levels, typologies, and geographical regions; and find that multilingual LLMs exhibit inconsistent cross-lingual alignments. To address this, we propose an interlingual representation framework identifying both the shared interlingual semantic subspace and fragmented components, existed due to representational limitations. We introduce Interlingual Local Overlap (ILO) score to quantify interlingual alignment by comparing the local neighborhood structures of high-dimensional representations. We utilize ILO to investigate the impact of single-language fine-tuning on the interlingual representations in multilingual LLMs. Our results indicate that training exclusively on a single language disrupts the alignment in early layers, while freezing these layers preserves the alignment of interlingual representations, leading to improved cross-lingual generalization. These results validate our framework and metric for evaluating interlingual representation, and further underscore that interlingual alignment is crucial for scalable multilingual learning.