Document Flattening: Beyond Concatenating Context for Document-Level Neural Machine Translation
This work addresses the problem of leveraging distant context in document-level translation for NLP researchers, representing an incremental improvement over existing methods.
The paper tackles the limitation of using concatenated pseudo-documents in document-level neural machine translation by introducing Document Flattening (DocFlat) with Flat-Batch Attention and Neural Context Gate, which outperforms strong baselines on BLEU, COMET, and accuracy metrics for English-German translation.
Existing work in document-level neural machine translation commonly concatenates several consecutive sentences as a pseudo-document, and then learns inter-sentential dependencies. This strategy limits the model's ability to leverage information from distant context. We overcome this limitation with a novel Document Flattening (DocFlat) technique that integrates Flat-Batch Attention (FBA) and Neural Context Gate (NCG) into Transformer model to utilize information beyond the pseudo-document boundaries. FBA allows the model to attend to all the positions in the batch and learns the relationships between positions explicitly and NCG identifies the useful information from the distant context. We conduct comprehensive experiments and analyses on three benchmark datasets for English-German translation, and validate the effectiveness of two variants of DocFlat. Empirical results show that our approach outperforms strong baselines with statistical significance on BLEU, COMET and accuracy on the contrastive test set. The analyses highlight that DocFlat is highly effective in capturing the long-range information.