Self-supervised Graph Masking Pre-training for Graph-to-Text Generation
This work improves Graph-to-Text generation for applications needing structured data conversion, but it is incremental as it builds on existing pre-trained models without architectural changes.
The paper tackled the problem of Graph-to-Text generation by addressing structural information loss and domain mismatch in pre-trained language models, achieving new state-of-the-art results on WebNLG+2020 and EventNarrative datasets.
Large-scale pre-trained language models (PLMs) have advanced Graph-to-Text (G2T) generation by processing the linearised version of a graph. However, the linearisation is known to ignore the structural information. Additionally, PLMs are typically pre-trained on free text which introduces domain mismatch between pre-training and downstream G2T generation tasks. To address these shortcomings, we propose graph masking pre-training strategies that neither require supervision signals nor adjust the architecture of the underlying pre-trained encoder-decoder model. When used with a pre-trained T5, our approach achieves new state-of-the-art results on WebNLG+2020 and EventNarrative G2T generation datasets. Our method also shows to be very effective in the low-resource setting.