No Train but Gain: Language Arithmetic for training-free Language Adapters enhancement
This work addresses a specific limitation in multilingual NLP for researchers and practitioners, offering an incremental enhancement to existing modular approaches.
The paper tackles the problem of reduced positive transfer between closely related languages in modular multilingual models by introducing language arithmetic, a training-free post-processing method that improves cross-lingual performance, achieving significant gains in zero-shot applications.
Modular deep learning is the state-of-the-art solution for lifting the curse of multilinguality, preventing the impact of negative interference and enabling cross-lingual performance in Multilingual Pre-trained Language Models. However, a trade-off of this approach is the reduction in positive transfer learning from closely related languages. In response, we introduce a novel method called language arithmetic, which enables training-free post-processing to address this limitation. Extending the task arithmetic framework, we apply learning via addition to the language adapters, transitioning the framework from a multi-task to a multilingual setup. The effectiveness of the proposed solution is demonstrated on three downstream tasks in a MAD-X-based set of cross-lingual schemes, acting as a post-processing procedure. Language arithmetic consistently improves the baselines with significant gains, especially in the most challenging case of zero-shot application. Our code and models are available at https://github.com/mklimasz/language-arithmetic .