Cross-Lingual Natural Language Generation via Pre-Training
This work addresses the challenge of cross-lingual natural language generation for low-resource languages, offering an incremental improvement over existing methods by using pre-training to reduce reliance on machine translation pipelines.
The paper tackles the problem of transferring supervision signals for natural language generation tasks across multiple languages by pre-training a sequence-to-sequence model in a shared space, enabling zero-shot cross-lingual transfer without machine translation. Experimental results show it outperforms machine-translation-based methods on tasks like question generation and abstractive summarization, improving performance for low-resource languages by leveraging data from rich-resource languages.
In this work we focus on transferring supervision signals of natural language generation (NLG) tasks between multiple languages. We propose to pretrain the encoder and the decoder of a sequence-to-sequence model under both monolingual and cross-lingual settings. The pre-training objective encourages the model to represent different languages in the shared space, so that we can conduct zero-shot cross-lingual transfer. After the pre-training procedure, we use monolingual data to fine-tune the pre-trained model on downstream NLG tasks. Then the sequence-to-sequence model trained in a single language can be directly evaluated beyond that language (i.e., accepting multi-lingual input and producing multi-lingual output). Experimental results on question generation and abstractive summarization show that our model outperforms the machine-translation-based pipeline methods for zero-shot cross-lingual generation. Moreover, cross-lingual transfer improves NLG performance of low-resource languages by leveraging rich-resource language data. Our implementation and data are available at https://github.com/CZWin32768/xnlg.