Diverse Text Generation via Variational Encoder-Decoder Models with Gaussian Process Priors
This work addresses the need for more diverse text generation in NLG applications, offering a novel approach that is incremental over existing encoder-decoder methods.
The authors tackled the problem of generating diverse, high-quality text by introducing a variational encoder-decoder model with Gaussian process priors, which improved both quality and diversity in paraphrase generation and text style transfer tasks compared to baselines.
Generating high quality texts with high diversity is important for many NLG applications, but current methods mostly focus on building deterministic models to generate higher quality texts and do not provide many options for promoting diversity. In this work, we present a novel latent structured variable model to generate high quality texts by enriching contextual representation learning of encoder-decoder models. Specifically, we introduce a stochastic function to map deterministic encoder hidden states into random context variables. The proposed stochastic function is sampled from a Gaussian process prior to (1) provide infinite number of joint Gaussian distributions of random context variables (diversity-promoting) and (2) explicitly model dependency between context variables (accurate-encoding). To address the learning challenge of Gaussian processes, we propose an efficient variational inference approach to approximate the posterior distribution of random context variables. We evaluate our method in two typical text generation tasks: paraphrase generation and text style transfer. Experimental results on benchmark datasets demonstrate that our method improves the generation quality and diversity compared with other baselines.