A 'Gibbs-Newton' Technique for Enhanced Inference of Multivariate Polya Parameters and Topic Models
This work addresses hyper-parameter tuning in topic modeling, an incremental improvement for researchers and practitioners in natural language processing.
The paper tackles the problem of hyper-parameter selection in latent Dirichlet allocation (LDA) by proposing LDA-GN, which uses non-informative priors and a new 'Gibbs-Newton' algorithm to learn these parameters, resulting in improved generalization to unseen documents and performance on a binary classification task compared to standard LDA.
Hyper-parameters play a major role in the learning and inference process of latent Dirichlet allocation (LDA). In order to begin the LDA latent variables learning process, these hyper-parameters values need to be pre-determined. We propose an extension for LDA that we call 'Latent Dirichlet allocation Gibbs Newton' (LDA-GN), which places non-informative priors over these hyper-parameters and uses Gibbs sampling to learn appropriate values for them. At the heart of LDA-GN is our proposed 'Gibbs-Newton' algorithm, which is a new technique for learning the parameters of multivariate Polya distributions. We report Gibbs-Newton performance results compared with two prominent existing approaches to the latter task: Minka's fixed-point iteration method and the Moments method. We then evaluate LDA-GN in two ways: (i) by comparing it with standard LDA in terms of the ability of the resulting topic models to generalize to unseen documents; (ii) by comparing it with standard LDA in its performance on a binary classification task.