Ordering-sensitive and Semantic-aware Topic Modeling
This work addresses the challenge of improving topic modeling accuracy for text analysis by incorporating ordering and semantics, representing an incremental advancement over existing methods.
The authors tackled the problem of topic modeling by addressing the limitations of the bag-of-words assumption, which ignores word ordering and semantics, and introduced the Gaussian Mixture Neural Topic Model (GMNTM) that incorporates both factors, resulting in significantly better performance in perplexity, retrieval accuracy, and classification accuracy compared to state-of-the-art methods.
Topic modeling of textual corpora is an important and challenging problem. In most previous work, the "bag-of-words" assumption is usually made which ignores the ordering of words. This assumption simplifies the computation, but it unrealistically loses the ordering information and the semantic of words in the context. In this paper, we present a Gaussian Mixture Neural Topic Model (GMNTM) which incorporates both the ordering of words and the semantic meaning of sentences into topic modeling. Specifically, we represent each topic as a cluster of multi-dimensional vectors and embed the corpus into a collection of vectors generated by the Gaussian mixture model. Each word is affected not only by its topic, but also by the embedding vector of its surrounding words and the context. The Gaussian mixture components and the topic of documents, sentences and words can be learnt jointly. Extensive experiments show that our model can learn better topics and more accurate word distributions for each topic. Quantitatively, comparing to state-of-the-art topic modeling approaches, GMNTM obtains significantly better performance in terms of perplexity, retrieval accuracy and classification accuracy.