End-to-end Learning of LDA by Mirror-Descent Back Propagation over a Deep Architecture
This work provides a scalable and end-to-end method for supervised topic modeling, addressing efficiency and performance issues for applications in text analysis, though it is incremental as it builds on existing LDA and deep learning techniques.
The authors tackled the problem of supervised Latent Dirichlet Allocation (LDA) by developing a fully discriminative learning approach using mirror-descent back propagation, which significantly outperformed previous supervised topic models and neural networks on three real-world tasks.
We develop a fully discriminative learning approach for supervised Latent Dirichlet Allocation (LDA) model using Back Propagation (i.e., BP-sLDA), which maximizes the posterior probability of the prediction variable given the input document. Different from traditional variational learning or Gibbs sampling approaches, the proposed learning method applies (i) the mirror descent algorithm for maximum a posterior inference and (ii) back propagation over a deep architecture together with stochastic gradient/mirror descent for model parameter estimation, leading to scalable and end-to-end discriminative learning of the model. As a byproduct, we also apply this technique to develop a new learning method for the traditional unsupervised LDA model (i.e., BP-LDA). Experimental results on three real-world regression and classification tasks show that the proposed methods significantly outperform the previous supervised topic models, neural networks, and is on par with deep neural networks.