Explaining Away Syntactic Structure in Semantic Document Representations
This work addresses the challenge of preserving syntactic structure in semantic document modeling for natural language processing applications, representing an incremental improvement over existing methods.
The authors tackled the problem of generative document models losing syntactic information by proposing a sequence-aware variational autoencoder that separates local syntactic context from global semantic representations, resulting in stronger topicality and increased robustness to syntactic noise in experiments.
Most generative document models act on bag-of-words input in an attempt to focus on the semantic content and thereby partially forego syntactic information. We argue that it is preferable to keep the original word order intact and explicitly account for the syntactic structure instead. We propose an extension to the Neural Variational Document Model (Miao et al., 2016) that does exactly that to separate local (syntactic) context from the global (semantic) representation of the document. Our model builds on the variational autoencoder framework to define a generative document model based on next-word prediction. We name our approach Sequence-Aware Variational Autoencoder since in contrast to its predecessor, it operates on the true input sequence. In a series of experiments we observe stronger topicality of the learned representations as well as increased robustness to syntactic noise in our training data.