Condensing Multilingual Knowledge with Lightweight Language-Specific Modules
This work addresses the problem of managing hundreds of languages in machine translation for researchers and practitioners, offering a more efficient approach with incremental improvements in parameter reduction and performance.
The paper tackles the scalability issue of language-specific modules in multilingual machine translation by introducing the Language-Specific Matrix Synthesis (LMS) method, which uses low-rank matrices to reduce parameters and achieves a 1.73 BLEU point improvement over the Switch Transformer.
Incorporating language-specific (LS) modules is a proven method to boost performance in multilingual machine translation. This approach bears similarity to Mixture-of-Experts (MoE) because it does not inflate FLOPs. However, the scalability of this approach to hundreds of languages (experts) tends to be unmanageable due to the prohibitive number of parameters introduced by full-rank matrices in fully-connected layers. In this work, we introduce the Language-Specific Matrix Synthesis (LMS) method. This approach constructs LS modules by generating low-rank matrices from two significantly smaller matrices to approximate the full-rank matrix. Furthermore, we condense multilingual knowledge from multiple LS modules into a single shared module with the Fuse Distillation (FD) technique to improve the efficiency of inference and model serialization. We show that our LMS method significantly outperforms previous LS methods and MoE methods with the same amount of extra parameters, e.g., 1.73 BLEU points over the Switch Transformer on many-to-many multilingual machine translation. Importantly, LMS is able to have comparable translation performance with much fewer parameters.