Neural Topic Modeling with Continual Lifelong Learning
This work addresses the problem of data sparsity in topic modeling for researchers and practitioners in natural language processing, offering an incremental improvement through a novel lifelong learning approach.
The paper tackles the challenge of applying topic modeling to sparse document collections by proposing a lifelong learning framework that continuously processes document streams, accumulating and transferring knowledge to improve topic coherence and document representations, demonstrating improved performance in perplexity, topic coherence, and information retrieval tasks.
Lifelong learning has recently attracted attention in building machine learning systems that continually accumulate and transfer knowledge to help future learning. Unsupervised topic modeling has been popularly used to discover topics from document collections. However, the application of topic modeling is challenging due to data sparsity, e.g., in a small collection of (short) documents and thus, generate incoherent topics and sub-optimal document representations. To address the problem, we propose a lifelong learning framework for neural topic modeling that can continuously process streams of document collections, accumulate topics and guide future topic modeling tasks by knowledge transfer from several sources to better deal with the sparse data. In the lifelong process, we particularly investigate jointly: (1) sharing generative homologies (latent topics) over lifetime to transfer prior knowledge, and (2) minimizing catastrophic forgetting to retain the past learning via novel selective data augmentation, co-training and topic regularization approaches. Given a stream of document collections, we apply the proposed Lifelong Neural Topic Modeling (LNTM) framework in modeling three sparse document collections as future tasks and demonstrate improved performance quantified by perplexity, topic coherence and information retrieval task.