Improving the Transformer Translation Model with Document-Level Context
This work addresses translation quality for languages with discourse phenomena, but it is incremental as it builds on the existing Transformer model.
The authors tackled the problem of incorporating document-level context into the Transformer translation model to address discourse issues, resulting in significant improvements over the baseline Transformer on NIST Chinese-English and IWSLT French-English datasets.
Although the Transformer translation model (Vaswani et al., 2017) has achieved state-of-the-art performance in a variety of translation tasks, how to use document-level context to deal with discourse phenomena problematic for Transformer still remains a challenge. In this work, we extend the Transformer model with a new context encoder to represent document-level context, which is then incorporated into the original encoder and decoder. As large-scale document-level parallel corpora are usually not available, we introduce a two-step training method to take full advantage of abundant sentence-level parallel corpora and limited document-level parallel corpora. Experiments on the NIST Chinese-English datasets and the IWSLT French-English datasets show that our approach improves over Transformer significantly.