Parallel Corpus Filtering via Pre-trained Language Models
This addresses the challenge of data quality for machine translation researchers and practitioners, offering an incremental improvement over existing filtering methods.
The paper tackles the problem of filtering noisy sentence pairs from web-crawled parallel corpora for machine translation by using pre-trained language models like BERT and GPT, achieving a new state-of-the-art on the WMT 2018 task and comparable performance to top supervised methods in an unsupervised setting.
Web-crawled data provides a good source of parallel corpora for training machine translation models. It is automatically obtained, but extremely noisy, and recent work shows that neural machine translation systems are more sensitive to noise than traditional statistical machine translation methods. In this paper, we propose a novel approach to filter out noisy sentence pairs from web-crawled corpora via pre-trained language models. We measure sentence parallelism by leveraging the multilingual capability of BERT and use the Generative Pre-training (GPT) language model as a domain filter to balance data domains. We evaluate the proposed method on the WMT 2018 Parallel Corpus Filtering shared task, and on our own web-crawled Japanese-Chinese parallel corpus. Our method significantly outperforms baselines and achieves a new state-of-the-art. In an unsupervised setting, our method achieves comparable performance to the top-1 supervised method. We also evaluate on a web-crawled Japanese-Chinese parallel corpus that we make publicly available.