Recovering document annotations for sentence-level bitext
This work addresses the data limitation problem for researchers and practitioners in machine translation by providing a new dataset and filtering method, though it is incremental as it builds on existing sentence-level paradigms.
The authors tackled the lack of document-level datasets for context-aware machine translation by reconstructing document-level information for three large datasets across multiple languages, and introduced a document-level filtering technique that improves document-level translation without degrading sentence-level performance.
Data availability limits the scope of any given task. In machine translation, historical models were incapable of handling longer contexts, so the lack of document-level datasets was less noticeable. Now, despite the emergence of long-sequence methods, we remain within a sentence-level paradigm and without data to adequately approach context-aware machine translation. Most large-scale datasets have been processed through a pipeline that discards document-level metadata. In this work, we reconstruct document-level information for three (ParaCrawl, News Commentary, and Europarl) large datasets in German, French, Spanish, Italian, Polish, and Portuguese (paired with English). We then introduce a document-level filtering technique as an alternative to traditional bitext filtering. We present this filtering with analysis to show that this method prefers context-consistent translations rather than those that may have been sentence-level machine translated. Last we train models on these longer contexts and demonstrate improvement in document-level translation without degradation of sentence-level translation. We release our dataset, ParaDocs, and resulting models as a resource to the community.