A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations
This work addresses the challenge of separating syntactic and semantic information in NLP, which is incremental as it builds on existing methods with specific improvements.
The authors tackled the problem of disentangling syntax and semantics in sentence representations by proposing a generative model with two latent variables, achieving better disentanglement through multi-task training with aligned paraphrases and word-order losses, and found that the best-performing model also yielded the most disentangled representations.
We propose a generative model for a sentence that uses two latent variables, with one intended to represent the syntax of the sentence and the other to represent its semantics. We show we can achieve better disentanglement between semantic and syntactic representations by training with multiple losses, including losses that exploit aligned paraphrastic sentences and word-order information. We also investigate the effect of moving from bag-of-words to recurrent neural network modules. We evaluate our models as well as several popular pretrained embeddings on standard semantic similarity tasks and novel syntactic similarity tasks. Empirically, we find that the model with the best performing syntactic and semantic representations also gives rise to the most disentangled representations.