DTMT: A Novel Deep Transition Architecture for Neural Machine Translation
This work addresses translation quality for machine translation systems, offering a novel architectural improvement that is incremental relative to existing RNN-based models.
The authors tackled the problem of shallow transition depth in RNN-based neural machine translation by introducing DTMT, a deep transition architecture that enhances hidden-to-hidden transitions with multiple non-linear transformations and a linear path to mitigate gradient vanishing, achieving improvements such as +2.09 BLEU points over Transformer on Chinese->English translation and superior results on WMT14 tasks.
Past years have witnessed rapid developments in Neural Machine Translation (NMT). Most recently, with advanced modeling and training techniques, the RNN-based NMT (RNMT) has shown its potential strength, even compared with the well-known Transformer (self-attentional) model. Although the RNMT model can possess very deep architectures through stacking layers, the transition depth between consecutive hidden states along the sequential axis is still shallow. In this paper, we further enhance the RNN-based NMT through increasing the transition depth between consecutive hidden states and build a novel Deep Transition RNN-based Architecture for Neural Machine Translation, named DTMT. This model enhances the hidden-to-hidden transition with multiple non-linear transformations, as well as maintains a linear transformation path throughout this deep transition by the well-designed linear transformation mechanism to alleviate the gradient vanishing problem. Experiments show that with the specially designed deep transition modules, our DTMT can achieve remarkable improvements on translation quality. Experimental results on Chinese->English translation task show that DTMT can outperform the Transformer model by +2.09 BLEU points and achieve the best results ever reported in the same dataset. On WMT14 English->German and English->French translation tasks, DTMT shows superior quality to the state-of-the-art NMT systems, including the Transformer and the RNMT+.