A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translation
This work addresses the problem of suboptimal subword configuration selection for neural machine translation practitioners, offering guidance to improve performance, though it is incremental as it builds on existing BPE methods.
The paper systematically explores how the number of Byte-Pair Encoding (BPE) merge operations affects neural machine translation performance, finding that sub-optimal configurations can reduce BLEU scores by 3-4 points, with LSTM architectures requiring varied settings and Transformers favoring smaller sizes.
Most neural machine translation systems are built upon subword units extracted by methods such as Byte-Pair Encoding (BPE) or wordpiece. However, the choice of number of merge operations is generally made by following existing recipes. In this paper, we conduct a systematic exploration on different numbers of BPE merge operations to understand how it interacts with the model architecture, the strategy to build vocabularies and the language pair. Our exploration could provide guidance for selecting proper BPE configurations in the future. Most prominently: we show that for LSTM-based architectures, it is necessary to experiment with a wide range of different BPE operations as there is no typical optimal BPE configuration, whereas for Transformer architectures, smaller BPE size tends to be a typically optimal choice. We urge the community to make prudent choices with subword merge operations, as our experiments indicate that a sub-optimal BPE configuration alone could easily reduce the system performance by 3-4 BLEU points.