Neural Machine Translation with Error Correction
This addresses translation accuracy issues in NMT for language processing applications, but it is incremental as it builds on existing methods like Transformer and XLNet.
The paper tackles the discrepancy between training and inference in neural machine translation (NMT) that leads to error propagation, by introducing an error correction mechanism using two-stream self-attention from XLNet and scheduled sampling, achieving improvements over Transformer baseline and scheduled sampling on multiple translation datasets.
Neural machine translation (NMT) generates the next target token given as input the previous ground truth target tokens during training while the previous generated target tokens during inference, which causes discrepancy between training and inference as well as error propagation, and affects the translation accuracy. In this paper, we introduce an error correction mechanism into NMT, which corrects the error information in the previous generated tokens to better predict the next token. Specifically, we introduce two-stream self-attention from XLNet into NMT decoder, where the query stream is used to predict the next token, and meanwhile the content stream is used to correct the error information from the previous predicted tokens. We leverage scheduled sampling to simulate the prediction errors during training. Experiments on three IWSLT translation datasets and two WMT translation datasets demonstrate that our method achieves improvements over Transformer baseline and scheduled sampling. Further experimental analyses also verify the effectiveness of our proposed error correction mechanism to improve the translation quality.