Let the Pretrained Language Models "Imagine" for Short Texts Topic Modeling
This addresses the challenge of topic modeling for short texts, a common issue in domains like social media or messaging, by leveraging pre-trained models to enhance data representation, though it is incremental as it builds on existing neural topic models and PLMs.
The paper tackles the problem of topic modeling for short texts, which suffer from data sparsity, by using pre-trained language models to extend short texts into longer sequences and incorporating a neural topic model to reduce noise. The result is a model that substantially improves performance, generating high-quality topics and outperforming state-of-the-art models in experiments on multiple real-world datasets under extreme sparsity scenarios.
Topic models are one of the compelling methods for discovering latent semantics in a document collection. However, it assumes that a document has sufficient co-occurrence information to be effective. However, in short texts, co-occurrence information is minimal, which results in feature sparsity in document representation. Therefore, existing topic models (probabilistic or neural) mostly fail to mine patterns from them to generate coherent topics. In this paper, we take a new approach to short-text topic modeling to address the data-sparsity issue by extending short text into longer sequences using existing pre-trained language models (PLMs). Besides, we provide a simple solution extending a neural topic model to reduce the effect of noisy out-of-topics text generation from PLMs. We observe that our model can substantially improve the performance of short-text topic modeling. Extensive experiments on multiple real-world datasets under extreme data sparsity scenarios show that our models can generate high-quality topics outperforming state-of-the-art models.