Diverse Pretrained Context Encodings Improve Document Translation
This work addresses the challenge of enhancing translation quality and sample efficiency for document-level machine translation tasks, representing an incremental improvement over prior methods.
The paper tackled the problem of improving document-level machine translation by incorporating multiple pretrained document context signals into a sentence-level transformer, resulting in a multi-context model that consistently outperforms existing context-aware transformers.
We propose a new architecture for adapting a sentence-level sequence-to-sequence transformer by incorporating multiple pretrained document context signals and assess the impact on translation performance of (1) different pretraining approaches for generating these signals, (2) the quantity of parallel data for which document context is available, and (3) conditioning on source, target, or source and target contexts. Experiments on the NIST Chinese-English, and IWSLT and WMT English-German tasks support four general conclusions: that using pretrained context representations markedly improves sample efficiency, that adequate parallel data resources are crucial for learning to use document context, that jointly conditioning on multiple context representations outperforms any single representation, and that source context is more valuable for translation performance than target side context. Our best multi-context model consistently outperforms the best existing context-aware transformers.