Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs
This addresses the challenge of making fine-tuned LLMs accessible across diverse languages, particularly benefiting lower-resource language communities, though it is incremental as it builds on existing fine-tuning methods.
The paper tackles the problem of cross-lingual transfer in fine-tuned LLMs by analyzing internal representations and proposing a middle-layer alignment objective, resulting in consistent improvements in tasks like slot filling and machine translation, especially for lower-resource languages.
While large language models demonstrate remarkable capabilities at task-specific applications through fine-tuning, extending these benefits across diverse languages is essential for broad accessibility. However, effective cross-lingual transfer is hindered by LLM performance gaps across languages and the scarcity of fine-tuning data in many languages. Through analysis of LLM internal representations from over 1,000+ language pairs, we discover that middle layers exhibit the strongest potential for cross-lingual alignment. Building on this finding, we propose a middle-layer alignment objective integrated into task-specific training. Our experiments on slot filling, machine translation, and structured text generation show consistent improvements in cross-lingual transfer, especially to lower-resource languages. The method is robust to the choice of alignment languages and generalizes to languages unseen during alignment. Furthermore, we show that separately trained alignment modules can be merged with existing task-specific modules, improving cross-lingual capabilities without full re-training. Our code is publicly available (https://github.com/dannigt/mid-align).