Multi-layer Representation Fusion for Neural Machine Translation
This work addresses a bottleneck in deep neural machine translation models for improving translation quality, though it is incremental as it builds on existing Transformer architectures.
The paper tackles the problem of information loss in neural machine translation by proposing a multi-layer representation fusion approach, which improves BLEU scores by 0.92 and 0.56 points on German-English and Chinese-English tasks, achieving new state-of-the-art in German-English translation.
Neural machine translation systems require a number of stacked layers for deep models. But the prediction depends on the sentence representation of the top-most layer with no access to low-level representations. This makes it more difficult to train the model and poses a risk of information loss to prediction. In this paper, we propose a multi-layer representation fusion (MLRF) approach to fusing stacked layers. In particular, we design three fusion functions to learn a better representation from the stack. Experimental results show that our approach yields improvements of 0.92 and 0.56 BLEU points over the strong Transformer baseline on IWSLT German-English and NIST Chinese-English MT tasks respectively. The result is new state-of-the-art in German-English translation.