Disentangling semantics in language through VAEs and a certain architectural choice
This work addresses the problem of obtaining disentangled semantic representations for natural language processing researchers, offering a new way to control and understand sentence structure.
This paper introduces an unsupervised method that uses a Variational Autoencoder with modified Transformers to disentangle semantic content in sentences. The model successfully separated verbs, subjects, direct objects, and prepositional objects into distinct latent variables, allowing for targeted manipulation and swapping of these semantic elements.
We present an unsupervised method to obtain disentangled representations of sentences that single out semantic content. Using modified Transformers as building blocks, we train a Variational Autoencoder to translate the sentence to a fixed number of hierarchically structured latent variables. We study the influence of each latent variable in generation on the dependency structure of sentences, and on the predicate structure it yields when passed through an Open Information Extraction model. Our model could separate verbs, subjects, direct objects, and prepositional objects into latent variables we identified. We show that varying the corresponding latent variables results in varying these elements in sentences, and that swapping them between couples of sentences leads to the expected partial semantic swap.