GradSim: Gradient-Based Language Grouping for Effective Multilingual Training
This addresses the challenge of effective multilingual training for low-resource languages, offering an incremental improvement in language grouping methods.
The paper tackles the problem of selecting compatible languages for multilingual training to avoid negative interference, proposing GradSim, a gradient-based language grouping method that achieves state-of-the-art performance on the AfriSenti benchmark for sentiment analysis in low-resource African languages.
Most languages of the world pose low-resource challenges to natural language processing models. With multilingual training, knowledge can be shared among languages. However, not all languages positively influence each other and it is an open research question how to select the most suitable set of languages for multilingual training and avoid negative interference among languages whose characteristics or data distributions are not compatible. In this paper, we propose GradSim, a language grouping method based on gradient similarity. Our experiments on three diverse multilingual benchmark datasets show that it leads to the largest performance gains compared to other similarity measures and it is better correlated with cross-lingual model performance. As a result, we set the new state of the art on AfriSenti, a benchmark dataset for sentiment analysis on low-resource African languages. In our extensive analysis, we further reveal that besides linguistic features, the topics of the datasets play an important role for language grouping and that lower layers of transformer models encode language-specific features while higher layers capture task-specific information.