Incremental Variational Inference for Latent Dirichlet Allocation
This work addresses scalability issues in topic modeling for large datasets, offering an incremental method that is faster and more stable, though it is incremental/hybrid in nature.
The paper tackles the problem of scaling latent Dirichlet allocation (LDA) to massive document collections by introducing incremental variational inference, which processes data efficiently without requiring a learning rate, converges faster, and monotonically increases the variational bound, with the distributed version achieving comparable performance and significant speed-up.
We introduce incremental variational inference and apply it to latent Dirichlet allocation (LDA). Incremental variational inference is inspired by incremental EM and provides an alternative to stochastic variational inference. Incremental LDA can process massive document collections, does not require to set a learning rate, converges faster to a local optimum of the variational bound and enjoys the attractive property of monotonically increasing it. We study the performance of incremental LDA on large benchmark data sets. We further introduce a stochastic approximation of incremental variational inference which extends to the asynchronous distributed setting. The resulting distributed algorithm achieves comparable performance as single host incremental variational inference, but with a significant speed-up.