Word, Subword or Character? An Empirical Study of Granularity in Chinese-English NMT
This addresses a domain-specific problem for machine translation researchers and practitioners by providing empirical guidance on granularity choices for Chinese-English NMT, which is incremental as it builds on existing granularity methods.
The study tackled the problem of selecting the optimal translation granularity (word, subword, or character) for Chinese-English neural machine translation, finding that subword models perform best for Chinese-to-English translation and hybrid word-character models are most suitable for English-to-Chinese translation, with the Hybrid_BPE method achieving the best results on Chinese-to-English tasks.
Neural machine translation (NMT), a new approach to machine translation, has been proved to outperform conventional statistical machine translation (SMT) across a variety of language pairs. Translation is an open-vocabulary problem, but most existing NMT systems operate with a fixed vocabulary, which causes the incapability of translating rare words. This problem can be alleviated by using different translation granularities, such as character, subword and hybrid word-character. Translation involving Chinese is one of the most difficult tasks in machine translation, however, to the best of our knowledge, there has not been any other work exploring which translation granularity is most suitable for Chinese in NMT. In this paper, we conduct an extensive comparison using Chinese-English NMT as a case study. Furthermore, we discuss the advantages and disadvantages of various translation granularities in detail. Our experiments show that subword model performs best for Chinese-to-English translation with the vocabulary which is not so big while hybrid word-character model is most suitable for English-to-Chinese translation. Moreover, experiments of different granularities show that Hybrid_BPE method can achieve best result on Chinese-to-English translation task.