JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs
This work improves text generation from knowledge graphs, which is important for applications like automated reporting or chatbots, but it is incremental as it builds on pre-trained models with novel modules and tasks.
The paper tackled the problem of generating text from knowledge graphs by addressing the neglect of graph structure and lack of explicit graph-text alignment in existing methods, resulting in new state-of-the-art performance on various datasets.
Existing pre-trained models for knowledge-graph-to-text (KG-to-text) generation simply fine-tune text-to-text pre-trained models such as BART or T5 on KG-to-text datasets, which largely ignore the graph structure during encoding and lack elaborate pre-training tasks to explicitly model graph-text alignments. To tackle these problems, we propose a graph-text joint representation learning model called JointGT. During encoding, we devise a structure-aware semantic aggregation module which is plugged into each Transformer layer to preserve the graph structure. Furthermore, we propose three new pre-training tasks to explicitly enhance the graph-text alignment including respective text / graph reconstruction, and graph-text alignment in the embedding space via Optimal Transport. Experiments show that JointGT obtains new state-of-the-art performance on various KG-to-text datasets.