Sparse Boltzmann Machines with Structure Learning as Applied to Text Analysis
This work addresses the need for more interpretable and efficient deep models in unsupervised text analysis, though it is incremental as it builds on existing RBM variants.
The paper tackled the problem of improving model fit and interpretability in Restricted Boltzmann Machines (RBMs) for text analysis by introducing Sparse Boltzmann Machines, where hidden units connect to subsets of visible units, resulting in significantly improved performance.
We are interested in exploring the possibility and benefits of structure learning for deep models. As the first step, this paper investigates the matter for Restricted Boltzmann Machines (RBMs). We conduct the study with Replicated Softmax, a variant of RBMs for unsupervised text analysis. We present a method for learning what we call Sparse Boltzmann Machines, where each hidden unit is connected to a subset of the visible units instead of all of them. Empirical results show that the method yields models with significantly improved model fit and interpretability as compared with RBMs where each hidden unit is connected to all visible units.