Beyond Shared Vocabulary: Increasing Representational Word Similarities across Languages for Multilingual Machine Translation
This work addresses a bottleneck in multilingual machine translation for languages with low lexical overlap, offering an incremental improvement over shared vocabulary methods.
The paper tackles the problem of limited knowledge transfer in multilingual machine translation when word overlap is small, especially across different writing systems, by defining word equivalence classes and using graph networks to fuse embeddings, resulting in consistent BLEU improvements of up to 2.3 points with minimal parameter and computational overhead.
Using a vocabulary that is shared across languages is common practice in Multilingual Neural Machine Translation (MNMT). In addition to its simple design, shared tokens play an important role in positive knowledge transfer, assuming that shared tokens refer to similar meanings across languages. However, when word overlap is small, especially due to different writing systems, transfer is inhibited. In this paper, we define word-level information transfer pathways via word equivalence classes and rely on graph networks to fuse word embeddings across languages. Our experiments demonstrate the advantages of our approach: 1) embeddings of words with similar meanings are better aligned across languages, 2) our method achieves consistent BLEU improvements of up to 2.3 points for high- and low-resource MNMT, and 3) less than 1.0\% additional trainable parameters are required with a limited increase in computational costs, while inference time remains identical to the baseline. We release the codebase to the community.