Registering Source Tokens to Target Language Spaces in Multilingual Neural Machine Translation
This work addresses the problem of low translation performance in MNMT models for researchers and developers, offering a novel approach that is incremental but shows strong gains.
The paper tackles the performance gap between multilingual neural machine translation (MNMT) models and large language models (LLMs) by introducing a 'registering' method that inserts target language tokens to improve translation accuracy, achieving state-of-the-art results on the EC-40 benchmark and outperforming NLLB-3.3B with their MITRE-913M model.
The multilingual neural machine translation (MNMT) aims for arbitrary translations across multiple languages. Although MNMT-specific models trained on parallel data offer low costs in training and deployment, their performance consistently lags behind that of large language models (LLMs). In this work, we introduce registering, a novel method that enables a small MNMT-specific model to compete with LLMs. Specifically, we insert a set of artificial tokens specifying the target language, called registers, into the input sequence between the source and target tokens. By modifying the attention mask, the target token generation only pays attention to the activation of registers, representing the source tokens in the target language space. Experiments on EC-40, a large-scale benchmark, show that our method advances the state-of-the-art of MNMT. We further pre-train two models, namely MITRE (multilingual translation with registers), by 9.3 billion sentence pairs across 24 languages collected from public corpora. One of them, MITRE-913M, outperforms NLLB-3.3B, achieves comparable performance with commercial LLMs, and shows strong adaptability in fine-tuning. Finally, we open-source our models to facilitate further research and development in MNMT: https://github.com/zhiqu22/mitre.