Efficiently Upgrading Multilingual Machine Translation Models to Support More Languages
This work addresses the incremental challenge of reusing and upgrading existing models to save computation as more language data becomes available, benefiting researchers and practitioners in machine translation.
The paper tackles the problem of efficiently upgrading multilingual machine translation models to support new languages by introducing techniques to speed up learning and reduce catastrophic forgetting, achieving results such as exceeding baseline performance with 30% computation and recovering larger model performance with over 50% reduction in computation.
With multilingual machine translation (MMT) models continuing to grow in size and number of supported languages, it is natural to reuse and upgrade existing models to save computation as data becomes available in more languages. However, adding new languages requires updating the vocabulary, which complicates the reuse of embeddings. The question of how to reuse existing models while also making architectural changes to provide capacity for both old and new languages has also not been closely studied. In this work, we introduce three techniques that help speed up effective learning of the new languages and alleviate catastrophic forgetting despite vocabulary and architecture mismatches. Our results show that by (1) carefully initializing the network, (2) applying learning rate scaling, and (3) performing data up-sampling, it is possible to exceed the performance of a same-sized baseline model with 30% computation and recover the performance of a larger model trained from scratch with over 50% reduction in computation. Furthermore, our analysis reveals that the introduced techniques help learn the new directions more effectively and alleviate catastrophic forgetting at the same time. We hope our work will guide research into more efficient approaches to growing languages for these MMT models and ultimately maximize the reuse of existing models.