Topic Modeling via Full Dependence Mixtures
This work addresses the challenge of efficient topic modeling for large-scale text data, representing an incremental improvement with a novel optimization method.
The authors tackled the problem of scaling topic modeling to large datasets by introducing the Full Dependence Mixture (FDM) model, which learns topics directly from co-occurrence data using a stochastic optimization procedure, and demonstrated that it performs comparably or better than standard benchmarks on datasets like NeurIPS papers and a Twitter corpus.
In this paper we introduce a new approach to topic modelling that scales to large datasets by using a compact representation of the data and by leveraging the GPU architecture. In this approach, topics are learned directly from the co-occurrence data of the corpus. In particular, we introduce a novel mixture model which we term the Full Dependence Mixture (FDM) model. FDMs model second moment under general generative assumptions on the data. While there is previous work on topic modeling using second moments, we develop a direct stochastic optimization procedure for fitting an FDM with a single Kullback Leibler objective. Moment methods in general have the benefit that an iteration no longer needs to scale with the size of the corpus. Our approach allows us to leverage standard optimizers and GPUs for the problem of topic modeling. In particular, we evaluate the approach on two large datasets, NeurIPS papers and a Twitter corpus, with a large number of topics, and show that the approach performs comparably or better than the the standard benchmarks.