How Does Pretraining Improve Discourse-Aware Translation?
This work provides insights into how discourse knowledge in pretrained models functions for downstream tasks like translation, but it is incremental as it focuses on analysis rather than new methods.
The paper tackled the problem of understanding why pretrained language models improve discourse-aware neural machine translation by introducing a probing task to interpret their ability to capture discourse relations, revealing that performance varies by architecture and layer and that discourse elements affect learning difficulty, with experiments on IWSLT2017 Chinese-English dataset showing NMT models initialized from different PLM layers follow similar trends.
Pretrained language models (PLMs) have produced substantial improvements in discourse-aware neural machine translation (NMT), for example, improved coherence in spoken language translation. However, the underlying reasons for their strong performance have not been well explained. To bridge this gap, we introduce a probing task to interpret the ability of PLMs to capture discourse relation knowledge. We validate three state-of-the-art PLMs across encoder-, decoder-, and encoder-decoder-based models. The analysis shows that (1) the ability of PLMs on discourse modelling varies from architecture and layer; (2) discourse elements in a text lead to different learning difficulties for PLMs. Besides, we investigate the effects of different PLMs on spoken language translation. Through experiments on IWSLT2017 Chinese-English dataset, we empirically reveal that NMT models initialized from different layers of PLMs exhibit the same trends with the probing task. Our findings are instructive to understand how and when discourse knowledge in PLMs should work for downstream tasks.