Improving Neural Topic Models using Knowledge Distillation
This work addresses the challenge of enhancing interpretable topic modeling for document analysis, though it appears incremental as it builds on existing methods with a modular framework.
The authors tackled the problem of improving topic quality in neural topic models by using knowledge distillation to combine probabilistic topic models and pretrained transformers, achieving state-of-the-art topic coherence with demonstrated improvements in both aggregate performance and head-to-head topic comparisons.
Topic models are often used to identify human-interpretable topics to help make sense of large document collections. We use knowledge distillation to combine the best attributes of probabilistic topic models and pretrained transformers. Our modular method can be straightforwardly applied with any neural topic model to improve topic quality, which we demonstrate using two models having disparate architectures, obtaining state-of-the-art topic coherence. We show that our adaptable framework not only improves performance in the aggregate over all estimated topics, as is commonly reported, but also in head-to-head comparisons of aligned topics.