Promoting Graph Awareness in Linearized Graph-to-Text Generation
This work addresses the problem of improving graph structure awareness in linearized graph-to-text generation models, which is significant for researchers and practitioners working with structured data and natural language generation, especially in data-scarce scenarios.
This paper investigates the ability of pretrained transformers, when applied to linearized graph inputs, to encode local graph structures. The authors found that using graph-denoising objectives as scaffolding in a multi-task text-to-text framework substantially improved downstream generation, particularly in low-resource settings.
Generating text from structured inputs, such as meaning representations or RDF triples, has often involved the use of specialized graph-encoding neural networks. However, recent applications of pretrained transformers to linearizations of graph inputs have yielded state-of-the-art generation results on graph-to-text tasks. Here, we explore the ability of these linearized models to encode local graph structures, in particular their invariance to the graph linearization strategy and their ability to reconstruct corrupted inputs. Our findings motivate solutions to enrich the quality of models' implicit graph encodings via scaffolding. Namely, we use graph-denoising objectives implemented in a multi-task text-to-text framework. We find that these denoising scaffolds lead to substantial improvements in downstream generation in low-resource settings.