Text Modeling with Syntax-Aware Variational Autoencoders
This work addresses the challenge of improving text modeling for natural language processing by integrating syntax, offering incremental advances in representation learning and generation.
The authors tackled the problem of unstructured latent representations in text VAEs by proposing syntax-aware VAEs (SAVAEs) that incorporate syntactic information into a dedicated latent subspace, achieving lower reconstruction loss on four datasets and enabling generation with modified target syntax.
Syntactic information contains structures and rules about how text sentences are arranged. Incorporating syntax into text modeling methods can potentially benefit both representation learning and generation. Variational autoencoders (VAEs) are deep generative models that provide a probabilistic way to describe observations in the latent space. When applied to text data, the latent representations are often unstructured. We propose syntax-aware variational autoencoders (SAVAEs) that dedicate a subspace in the latent dimensions dubbed syntactic latent to represent syntactic structures of sentences. SAVAEs are trained to infer syntactic latent from either text inputs or parsed syntax results as well as reconstruct original text with inferred latent variables. Experiments show that SAVAEs are able to achieve lower reconstruction loss on four different data sets. Furthermore, they are capable of generating examples with modified target syntax.