Efficient Learning for Undirected Topic Models
This work addresses the problem of slow training for topic models, which is incremental as it improves efficiency for researchers and practitioners in natural language processing.
The paper tackled the inefficiency of learning in undirected topic models like Replicated Softmax by introducing a novel estimator based on Noise Contrastive Estimation, which achieved great learning efficiency and high accuracy on document retrieval and classification tasks in experiments on two benchmarks.
Replicated Softmax model, a well-known undirected topic model, is powerful in extracting semantic representations of documents. Traditional learning strategies such as Contrastive Divergence are very inefficient. This paper provides a novel estimator to speed up the learning based on Noise Contrastive Estimate, extended for documents of variant lengths and weighted inputs. Experiments on two benchmarks show that the new estimator achieves great learning efficiency and high accuracy on document retrieval and classification.