Deep State Space Models for Unconditional Word Generation
This addresses the problem of systematic biases in autoregressive text generation for NLP researchers, offering an interpretable alternative.
The paper tackled unconditional word generation by proposing a non-autoregressive deep state space model that separates global and local uncertainty, achieving performance on par with autoregressive models.
Autoregressive feedback is considered a necessity for successful unconditional text generation using stochastic sequence models. However, such feedback is known to introduce systematic biases into the training process and it obscures a principle of generation: committing to global information and forgetting local nuances. We show that a non-autoregressive deep state space model with a clear separation of global and local uncertainty can be built from only two ingredients: An independent noise source and a deterministic transition function. Recent advances on flow-based variational inference can be used to train an evidence lower-bound without resorting to annealing, auxiliary losses or similar measures. The result is a highly interpretable generative model on par with comparable auto-regressive models on the task of word generation.