The Role of Language Imbalance in Cross-lingual Generalisation: Insights from Cloned Language Experiments
This addresses the problem of improving multilingual model efficiency for diverse linguistic communities, though it is incremental as it builds on prior work on parallel data and vocabulary.
The study investigated the role of language imbalance in cross-lingual generalization, finding that a predominant language during training boosts performance and representation alignment for less frequent languages, with a 90/10 split outperforming a 50/50 split in bilingual models at scale.
Multilinguality is crucial for extending recent advancements in language modelling to diverse linguistic communities. To maintain high performance while representing multiple languages, multilingual models ideally align representations, allowing what is learned in one language to generalise to others. Prior research has emphasised the importance of parallel data and shared vocabulary elements as key factors for such alignment. In this study, we investigate an unintuitive novel driver of cross-lingual generalisation: language imbalance. In controlled experiments on perfectly equivalent cloned languages, we observe that the existence of a predominant language during training boosts the performance of less frequent languages and leads to stronger alignment of model representations across languages. Furthermore, we find that this trend is amplified with scale: with large enough models or long enough training, we observe that bilingual training data with a 90/10 language split yields better performance on both languages than a balanced 50/50 split. Building on these insights, we design training schemes that can improve performance in all cloned languages, even without altering the training data. As we extend our analysis to real languages, we find that infrequent languages still benefit from frequent ones, yet whether language imbalance causes cross-lingual generalisation there is not conclusive.