Memory-Efficient Topic Modeling
This addresses memory constraints for researchers and practitioners working with large-scale topic modeling applications, such as text mining and computational biology, though it is an incremental improvement over existing message passing frameworks.
The paper tackled the high memory usage problem in training latent Dirichlet allocation (LDA) for topic modeling by proposing tiny belief propagation (TBP), which reduces memory consumption significantly, enabling topic modeling on massive corpora like a 7 GB PUBMED dataset with only 2GB of memory while performing comparably or better than state-of-the-art methods.
As one of the simplest probabilistic topic modeling techniques, latent Dirichlet allocation (LDA) has found many important applications in text mining, computer vision and computational biology. Recent training algorithms for LDA can be interpreted within a unified message passing framework. However, message passing requires storing previous messages with a large amount of memory space, increasing linearly with the number of documents or the number of topics. Therefore, the high memory usage is often a major problem for topic modeling of massive corpora containing a large number of topics. To reduce the space complexity, we propose a novel algorithm without storing previous messages for training LDA: tiny belief propagation (TBP). The basic idea of TBP relates the message passing algorithms with the non-negative matrix factorization (NMF) algorithms, which absorb the message updating into the message passing process, and thus avoid storing previous messages. Experimental results on four large data sets confirm that TBP performs comparably well or even better than current state-of-the-art training algorithms for LDA but with a much less memory consumption. TBP can do topic modeling when massive corpora cannot fit in the computer memory, for example, extracting thematic topics from 7 GB PUBMED corpora on a common desktop computer with 2GB memory.