A Post-trainer's Guide to Multilingual Training Data: Uncovering Cross-lingual Transfer Dynamics
This work addresses the challenge of making language models useful globally by providing insights into multilingual training, but it is incremental as it builds on existing post-training methods without introducing new paradigms.
This study tackled the problem of understanding cross-lingual transfer dynamics in multilingual post-training for large language models, finding that these dynamics vary with post-training settings and identifying conditions for effective transfer.
In order for large language models to be useful across the globe, they are fine-tuned to follow instructions on multilingual data. Despite the ubiquity of such post-training, a clear understanding of the dynamics that enable cross-lingual transfer remains elusive. This study examines cross-lingual transfer (CLT) dynamics in realistic post-training settings. We study two model families of up to 35B parameters in size trained on carefully controlled mixtures of multilingual data on three generative tasks with varying levels of complexity (summarization, instruction following, and mathematical reasoning) in both single-task and multi-task instruction tuning settings. Overall, we find that the dynamics of cross-lingual transfer and multilingual performance cannot be explained by isolated variables, varying depending on the combination of post-training settings. Finally, we identify the conditions that lead to effective cross-lingual transfer in practice.