Latent Template Induction with Gumbel-CRFs
This work offers a novel approach for researchers and practitioners in text generation to achieve better control over sentence structure, particularly in data-to-text and unsupervised paraphrase generation.
This paper addresses the challenge of controlling sentence structure in text generation by proposing a Gumbel-CRF, a continuous relaxation of the CRF sampling algorithm. This method enables the inference of latent templates, leading to more stable gradients and interpretable templates for controlling sentence generation.
Learning to control the structure of sentences is a challenging problem in text generation. Existing work either relies on simple deterministic approaches or RL-based hard structures. We explore the use of structured variational autoencoders to infer latent templates for sentence generation using a soft, continuous relaxation in order to utilize reparameterization for training. Specifically, we propose a Gumbel-CRF, a continuous relaxation of the CRF sampling algorithm using a relaxed Forward-Filtering Backward-Sampling (FFBS) approach. As a reparameterized gradient estimator, the Gumbel-CRF gives more stable gradients than score-function based estimators. As a structured inference network, we show that it learns interpretable templates during training, which allows us to control the decoder during testing. We demonstrate the effectiveness of our methods with experiments on data-to-text generation and unsupervised paraphrase generation.