Communication-Free Parallel Supervised Topic Models
This work addresses a specific bottleneck in parallelizing topic models for researchers and practitioners, but it is incremental as it adapts existing methods to a particular model.
The paper tackled the problem of applying communication-free parallel MCMC to supervised topic models by developing an algorithm for sLDA that overcomes quasi-ergodicity through reordering sampling steps, resulting in comparable out-of-sample prediction performance and significantly reduced computation time.
Embarrassingly (communication-free) parallel Markov chain Monte Carlo (MCMC) methods are commonly used in learning graphical models. However, MCMC cannot be directly applied in learning topic models because of the quasi-ergodicity problem caused by multimodal distribution of topics. In this paper, we develop an embarrassingly parallel MCMC algorithm for sLDA. Our algorithm works by switching the order of sampled topics combination and labeling variable prediction in sLDA, in which it overcomes the quasi-ergodicity problem because high-dimension topics that follow a multimodal distribution are projected into one-dimension document labels that follow a unimodal distribution. Our empirical experiments confirm that the out-of-sample prediction performance using our embarrassingly parallel algorithm is comparable to non-parallel sLDA while the computation time is significantly reduced.