Latent Normalizing Flows for Discrete Sequences
This work addresses the problem of efficient and flexible generative modeling for discrete sequences, such as text and music, for researchers and practitioners in machine learning, though it is incremental as it builds on existing VAE and flow methods.
The paper tackled the challenge of applying normalizing flows to discrete sequences like text by proposing a VAE-based model that learns a flow-based distribution in latent space and a stochastic mapping to observed discrete data, achieving performance matching autoregressive baselines in language modeling and music generation tasks.
Normalizing flows are a powerful class of generative models for continuous random variables, showing both strong model flexibility and the potential for non-autoregressive generation. These benefits are also desired when modeling discrete random variables such as text, but directly applying normalizing flows to discrete sequences poses significant additional challenges. We propose a VAE-based generative model which jointly learns a normalizing flow-based distribution in the latent space and a stochastic mapping to an observed discrete space. In this setting, we find that it is crucial for the flow-based distribution to be highly multimodal. To capture this property, we propose several normalizing flow architectures to maximize model flexibility. Experiments consider common discrete sequence tasks of character-level language modeling and polyphonic music generation. Our results indicate that an autoregressive flow-based model can match the performance of a comparable autoregressive baseline, and a non-autoregressive flow-based model can improve generation speed with a penalty to performance.