Modeling Documents with Deep Boltzmann Machines
This work improves document modeling for tasks like retrieval and classification, but it is incremental as it builds on existing DBM and RBM methods.
The authors tackled document modeling by introducing a Deep Boltzmann Machine with parameter tying for efficient training, achieving better log probability on unseen data than Replicated Softmax and outperforming LDA, Replicated Softmax, and DocNADE in retrieval and classification tasks.
We introduce a Deep Boltzmann Machine model suitable for modeling and extracting latent semantic representations from a large unstructured collection of documents. We overcome the apparent difficulty of training a DBM with judicious parameter tying. This parameter tying enables an efficient pretraining algorithm and a state initialization scheme that aids inference. The model can be trained just as efficiently as a standard Restricted Boltzmann Machine. Our experiments show that the model assigns better log probability to unseen data than the Replicated Softmax model. Features extracted from our model outperform LDA, Replicated Softmax, and DocNADE models on document retrieval and document classification tasks.