How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuning
This work addresses the problem of understanding cross-lingual mechanisms in MLLMs for researchers and practitioners, but it is incremental as it applies an existing method to a new perspective.
The study investigates how languages influence each other during multilingual fine-tuning of large language models by using TracIn to analyze cross-lingual data sharing at the data level, finding that models rely on multiple languages from early stages and this reliance increases over time, with fine-tuning languages reinforcing and complementing knowledge from the test language.
Multilingual large language models (MLLMs) are jointly trained on data from many different languages such that representation of individual languages can benefit from other languages' data. Impressive performance on zero-shot cross-lingual transfer shows that these models are capable of exploiting data from other languages. Yet, it remains unclear to what extent, and under which conditions, languages rely on each other's data. In this study, we use TracIn (Pruthi et al., 2020), a training data attribution (TDA) method, to retrieve the most influential training samples seen during multilingual fine-tuning for a particular test language. This allows us to analyse cross-lingual sharing mechanisms of MLLMs from a new perspective. While previous work studied cross-lingual sharing at the level of model parameters, we present the first approach to study cross-lingual sharing at the data level. We find that MLLMs rely on data from multiple languages from the early stages of fine-tuning and that this reliance gradually increases as fine-tuning progresses. We further study how different fine-tuning languages influence model performance on a given test language and find that they can both reinforce and complement the knowledge acquired from data of the test language itself.