Training Multilingual Machine Translation by Alternately Freezing Language-Specific Encoders-Decoders
This addresses the challenge of scalable multilingual translation for AI systems, but it is incremental as it builds on existing modular approaches with a novel training technique.
The authors tackled the problem of incrementally extending multilingual machine translation systems to new languages without retraining, by proposing a modular architecture with language-specific encoder-decoders trained via alternately freezing modules. The result showed successful training with improvements on initial languages, though performance slightly declined when adding new languages or in zero-shot translation, and benefits were also observed in natural language inference tasks.
We propose a modular architecture of language-specific encoder-decoders that constitutes a multilingual machine translation system that can be incrementally extended to new languages without the need for retraining the existing system when adding new languages. Differently from previous works, we simultaneously train $N$ languages in all translation directions by alternately freezing encoder or decoder modules, which indirectly forces the system to train in a common intermediate representation for all languages. Experimental results from multilingual machine translation show that we can successfully train this modular architecture improving on the initial languages while falling slightly behind when adding new languages or doing zero-shot translation. Additional comparison of the quality of sentence representation in the task of natural language inference shows that the alternately freezing training is also beneficial in this direction.