Topic Modeling with Contextualized Word Representation Clusters
This provides a simple, reliable method for topic modeling that handles polysemy better than traditional approaches, though it appears incremental as an adaptation of existing contextualized models.
The authors tackled the problem of topic modeling by clustering token-level contextualized word representations from models like BERT and GPT-2, finding that this approach captures polysemy better than vocabulary-level embeddings and performs comparably or better than LDA topic models, especially with many topics relative to collection size.
Clustering token-level contextualized word representations produces output that shares many similarities with topic models for English text collections. Unlike clusterings of vocabulary-level word embeddings, the resulting models more naturally capture polysemy and can be used as a way of organizing documents. We evaluate token clusterings trained from several different output layers of popular contextualized language models. We find that BERT and GPT-2 produce high quality clusterings, but RoBERTa does not. These cluster models are simple, reliable, and can perform as well as, if not better than, LDA topic models, maintaining high topic quality even when the number of topics is large relative to the size of the local collection.