Vocabulary Learning via Optimal Transport for Neural Machine Translation
This work provides a more efficient and effective way to select vocabularies for neural machine translation models, benefiting researchers and practitioners by reducing computational costs and improving translation quality.
This paper addresses the problem of finding an optimal token vocabulary for neural machine translation without extensive trial training. The authors propose VOLT, a method based on optimal transport, which achieves a 70% vocabulary size reduction and a 0.5 BLEU gain on English-German translation, while also reducing search time from 384 GPU hours to 30 GPU hours compared to BPE-search.
The choice of token vocabulary affects the performance of machine translation. This paper aims to figure out what is a good vocabulary and whether one can find the optimal vocabulary without trial training. To answer these questions, we first provide an alternative understanding of the role of vocabulary from the perspective of information theory. Motivated by this, we formulate the quest of vocabularization -- finding the best token dictionary with a proper size -- as an optimal transport (OT) problem. We propose VOLT, a simple and efficient solution without trial training. Empirical results show that VOLT outperforms widely-used vocabularies in diverse scenarios, including WMT-14 English-German and TED's 52 translation directions. For example, VOLT achieves almost 70% vocabulary size reduction and 0.5 BLEU gain on English-German translation. Also, compared to BPE-search, VOLT reduces the search time from 384 GPU hours to 30 GPU hours on English-German translation. Codes are available at https://github.com/Jingjing-NLP/VOLT .