Sparse Parallel Training of Hierarchical Dirichlet Process Topic Models
This enables efficient large-scale topic modeling for practitioners dealing with massive text corpora, though it is incremental as it builds on existing parallel and sparse methods.
The paper tackled scaling hierarchical Dirichlet process topic models to large datasets by proposing a doubly sparse data-parallel sampler that leverages sparsity in natural language, achieving training on 8 million documents with 768 million tokens in under four days on a single multi-core machine.
To scale non-parametric extensions of probabilistic topic models such as Latent Dirichlet allocation to larger data sets, practitioners rely increasingly on parallel and distributed systems. In this work, we study data-parallel training for the hierarchical Dirichlet process (HDP) topic model. Based upon a representation of certain conditional distributions within an HDP, we propose a doubly sparse data-parallel sampler for the HDP topic model. This sampler utilizes all available sources of sparsity found in natural language - an important way to make computation efficient. We benchmark our method on a well-known corpus (PubMed) with 8m documents and 768m tokens, using a single multi-core machine in under four days.