Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translation
This addresses the problem of scalable and efficient translation for 433 languages, offering a significant speedup and performance improvement, though it builds on classic multi-way structures.
The paper tackles parameter interference and inefficient inference in multilingual neural machine translation by developing Lego-MT, a detachable model with individual branches for languages, which achieves an average gain of 3.2 spBLEU and outperforms a larger model with 12B parameters.
Multilingual neural machine translation (MNMT) aims to build a unified model for many language directions. Existing monolithic models for MNMT encounter two challenges: parameter interference among languages and inefficient inference for large models. In this paper, we revisit the classic multi-way structures and develop a detachable model by assigning each language (or group of languages) to an individual branch that supports plug-and-play training and inference. To address the needs of learning representations for all languages in a unified space, we propose a novel efficient training recipe, upon which we build an effective detachable model, Lego-MT. For a fair comparison, we collect data from OPUS and build a translation benchmark covering 433 languages and 1.3B parallel data. Experiments show that Lego-MT with 1.2B parameters brings an average gain of 3.2 spBLEU. It even outperforms M2M-100 with 12B parameters. The proposed training recipe brings a 28.2$\times$ speedup over the conventional multi-way training method.\footnote{ \url{https://github.com/CONE-MT/Lego-MT}.}