Do Neural Topic Models Really Need Dropout? Analysis of the Effect of Dropout in Topic Modeling
This work addresses the lack of analysis on dropout for unsupervised models, specifically in neural topic modeling, which is incremental as it applies existing techniques to a new context.
The paper analyzed the effect of dropout regularization in VAE-based neural topic models, finding that it impacts topic quality and predictive performance across three models and four datasets.
Dropout is a widely used regularization trick to resolve the overfitting issue in large feedforward neural networks trained on a small dataset, which performs poorly on the held-out test subset. Although the effectiveness of this regularization trick has been extensively studied for convolutional neural networks, there is a lack of analysis of it for unsupervised models and in particular, VAE-based neural topic models. In this paper, we have analyzed the consequences of dropout in the encoder as well as in the decoder of the VAE architecture in three widely used neural topic models, namely, contextualized topic model (CTM), ProdLDA, and embedded topic model (ETM) using four publicly available datasets. We characterize the dropout effect on these models in terms of the quality and predictive performance of the generated topics.