Syntax-driven Iterative Expansion Language Models for Controllable Text Generation
This work addresses the need for more controllable and efficient text generation in natural language processing, though it is incremental as it builds on existing Transformer models.
The authors tackled the problem of sequential constraints in neural text generation by introducing a syntax-driven paradigm that uses dependency parse trees to guide Transformer models, resulting in generation quality between LSTMs and Transformers with comparable diversity, requiring less than half the decoding steps and enabling direct control over syntactic constructions for stylistic variations.
The dominant language modeling paradigm handles text as a sequence of discrete tokens. While that approach can capture the latent structure of the text, it is inherently constrained to sequential dynamics for text generation. We propose a new paradigm for introducing a syntactic inductive bias into neural text generation, where the dependency parse tree is used to drive the Transformer model to generate sentences iteratively. Our experiments show that this paradigm is effective at text generation, with quality between LSTMs and Transformers, and comparable diversity, requiring less than half their decoding steps, and its generation process allows direct control over the syntactic constructions of the generated text, enabling the induction of stylistic variations.