A Discrete Variational Recurrent Topic Model without the Reparametrization Trick
This work addresses a technical bottleneck in neural topic modeling for researchers and practitioners, though it appears incremental as it builds on existing neural variational inference methods.
The authors tackled the problem of learning neural topic models with discrete random variables without using stochastic backpropagation, achieving improved perplexity and document understanding across multiple corpora, with their approach competing and surpassing a popular topic model implementation on topic quality measures.
We show how to learn a neural topic model with discrete random variables---one that explicitly models each word's assigned topic---using neural variational inference that does not rely on stochastic backpropagation to handle the discrete variables. The model we utilize combines the expressive power of neural methods for representing sequences of text with the topic model's ability to capture global, thematic coherence. Using neural variational inference, we show improved perplexity and document understanding across multiple corpora. We examine the effect of prior parameters both on the model and variational parameters and demonstrate how our approach can compete and surpass a popular topic model implementation on an automatic measure of topic quality.