SenGen: Sentence Generating Neural Variational Topic Model
This work addresses the need for more interpretable topic modeling in natural language processing, though it appears incremental as it builds on existing variational auto-encoder methods with a sentence-level focus.
The paper tackles the problem of generating documents with interpretable topics by proposing a topic model that samples topics per sentence and uses an RNN decoder, aiming to improve discourse structure visualization and topic interpretability through representative sentences. Preliminary experiments on two datasets show early promise but highlight unresolved challenges.
We present a new topic model that generates documents by sampling a topic for one whole sentence at a time, and generating the words in the sentence using an RNN decoder that is conditioned on the topic of the sentence. We argue that this novel formalism will help us not only visualize and model the topical discourse structure in a document better, but also potentially lead to more interpretable topics since we can now illustrate topics by sampling representative sentences instead of bag of words or phrases. We present a variational auto-encoder approach for learning in which we use a factorized variational encoder that independently models the posterior over topical mixture vectors of documents using a feed-forward network, and the posterior over topic assignments to sentences using an RNN. Our preliminary experiments on two different datasets indicate early promise, but also expose many challenges that remain to be addressed.