DeTiME: Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLM
This addresses the problem of inefficient topic modeling and text generation for NLP researchers and practitioners, though it appears incremental as it combines existing techniques (LLMs and diffusion models) in a novel way.
The paper tackles the limitations of Neural Topic Models (NTMs) by introducing DeTiME, a framework that combines encoder-decoder LLMs with diffusion models to produce highly clusterable embeddings and enable topic-based text generation, achieving superior clusterability and semantic coherence compared to existing methods.
In the burgeoning field of natural language processing (NLP), Neural Topic Models (NTMs) , Large Language Models (LLMs) and Diffusion model have emerged as areas of significant research interest. Despite this, NTMs primarily utilize contextual embeddings from LLMs, which are not optimal for clustering or capable for topic based text generation. NTMs have never been combined with diffusion model for text generation. Our study addresses these gaps by introducing a novel framework named Diffusion-Enhanced Topic Modeling using Encoder-Decoder-based LLMs (DeTiME). DeTiME leverages Encoder-Decoder-based LLMs to produce highly clusterable embeddings that could generate topics that exhibit both superior clusterability and enhanced semantic coherence compared to existing methods. Additionally, by exploiting the power of diffusion model, our framework also provides the capability to do topic based text generation. This dual functionality allows users to efficiently produce highly clustered topics and topic based text generation simultaneously. DeTiME's potential extends to generating clustered embeddings as well. Notably, our proposed framework(both encoder-decoder based LLM and diffusion model) proves to be efficient to train and exhibits high adaptability to other LLMs and diffusion model, demonstrating its potential for a wide array of applications.