Document Informed Neural Autoregressive Topic Models
This work addresses the need for better topic modeling in natural language processing by improving interpretability and applicability across domains, though it is incremental as it builds on existing neural autoregressive models.
The paper tackles the problem of generative topic models not fully utilizing context information around words, by extending a neural autoregressive topic model to exploit full document context in a language modeling fashion, resulting in improved performance with a 9.6% gain in precision and 7.2% gain in F1 for text categorization.
Context information around words helps in determining their actual meaning, for example "networks" used in contexts of artificial neural networks or biological neuron networks. Generative topic models infer topic-word distributions, taking no or only little context into account. Here, we extend a neural autoregressive topic model to exploit the full context information around words in a document in a language modeling fashion. This results in an improved performance in terms of generalization, interpretability and applicability. We apply our modeling approach to seven data sets from various domains and demonstrate that our approach consistently outperforms stateof-the-art generative topic models. With the learned representations, we show on an average a gain of 9.6% (0.57 Vs 0.52) in precision at retrieval fraction 0.02 and 7.2% (0.582 Vs 0.543) in F1 for text categorization.