Disentangling Generative Factors in Natural Language with Discrete Variational Autoencoders
This work addresses the need for interpretable NLP systems by enabling control over latent linguistic features, though it is incremental as it builds on existing VAE frameworks with a discrete adaptation.
The authors tackled the problem of learning disentangled representations in natural language by arguing that continuous variables are suboptimal for modeling discrete textual features, and they proposed a discrete Variational Autoencoder method that outperformed baselines on disentanglement benchmarks and a style transfer task.
The ability of learning disentangled representations represents a major step for interpretable NLP systems as it allows latent linguistic features to be controlled. Most approaches to disentanglement rely on continuous variables, both for images and text. We argue that despite being suitable for image datasets, continuous variables may not be ideal to model features of textual data, due to the fact that most generative factors in text are discrete. We propose a Variational Autoencoder based method which models language features as discrete variables and encourages independence between variables for learning disentangled representations. The proposed model outperforms continuous and discrete baselines on several qualitative and quantitative benchmarks for disentanglement as well as on a text style transfer downstream application.