Gaussian Hierarchical Latent Dirichlet Allocation: Bringing Polysemy Back
This work addresses a key problem in topic modeling for researchers and practitioners by enhancing polysemy handling, though it is incremental as it builds on existing Gaussian and hierarchical LDA methods.
The paper tackles the limitation of Gaussian latent Dirichlet allocation in capturing word polysemy by introducing a hierarchical topic structure, resulting in significantly improved polysemy detection, more parsimonious topic representations, and better topic coherence and predictive accuracy across various corpora and word embeddings.
Topic models are widely used to discover the latent representation of a set of documents. The two canonical models are latent Dirichlet allocation, and Gaussian latent Dirichlet allocation, where the former uses multinomial distributions over words, and the latter uses multivariate Gaussian distributions over pre-trained word embedding vectors as the latent topic representations, respectively. Compared with latent Dirichlet allocation, Gaussian latent Dirichlet allocation is limited in the sense that it does not capture the polysemy of a word such as ``bank.'' In this paper, we show that Gaussian latent Dirichlet allocation could recover the ability to capture polysemy by introducing a hierarchical structure in the set of topics that the model can use to represent a given document. Our Gaussian hierarchical latent Dirichlet allocation significantly improves polysemy detection compared with Gaussian-based models and provides more parsimonious topic representations compared with hierarchical latent Dirichlet allocation. Our extensive quantitative experiments show that our model also achieves better topic coherence and held-out document predictive accuracy over a wide range of corpus and word embedding vectors.