Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment
This addresses the problem of uneven performance across languages in multilingual AI models, offering an incremental improvement for natural language processing applications.
The paper tackles the performance bias of multilingual generative models toward high-resource languages and isolated sentence representations by proposing a cross-lingual alignment framework using translation pairs and contrastive learning, which significantly boosts cross-lingual abilities and reduces the performance gap with less than 0.1‰ of pre-training tokens.
Multilingual generative models obtain remarkable cross-lingual in-context learning capabilities through pre-training on large-scale corpora. However, they still exhibit a performance bias toward high-resource languages and learn isolated distributions of multilingual sentence representations, which may hinder knowledge transfer across languages. To bridge this gap, we propose a simple yet effective cross-lingual alignment framework exploiting pairs of translation sentences. It aligns the internal sentence representations across different languages via multilingual contrastive learning and aligns outputs by following cross-lingual instructions in the target language. Experimental results show that even with less than 0.1 {\textperthousand} of pre-training tokens, our alignment framework significantly boosts the cross-lingual abilities of generative language models and mitigates the performance gap. Further analyses reveal that it results in a better internal multilingual representation distribution of multilingual models.