Examining Modularity in Multilingual LMs via Language-Specialized Subnetworks
This work addresses the problem of understanding and controlling cross-lingual interactions in multilingual models for NLP researchers, but it is incremental as it builds on prior modularity proposals.
The study investigated whether language-specialized subnetworks naturally occur in multilingual language models without interventions and compared cross-lingual sharing between such models and those with explicit modularity via sparse fine-tuning, finding that subnetworks do arise naturally and that sparse fine-tuning can reduce language specialization in favor of increased cross-lingual sharing.
Recent work has proposed explicitly inducing language-wise modularity in multilingual LMs via sparse fine-tuning (SFT) on per-language subnetworks as a means of better guiding cross-lingual sharing. In this work, we investigate (1) the degree to which language-wise modularity naturally arises within models with no special modularity interventions, and (2) how cross-lingual sharing and interference differ between such models and those with explicit SFT-guided subnetwork modularity. To quantify language specialization and cross-lingual interaction, we use a Training Data Attribution method that estimates the degree to which a model's predictions are influenced by in-language or cross-language training examples. Our results show that language-specialized subnetworks do naturally arise, and that SFT, rather than always increasing modularity, can decrease language specialization of subnetworks in favor of more cross-lingual sharing.