A derivation of variational message passing (VMP) for latent Dirichlet allocation (LDA)
This work addresses a technical bottleneck for researchers implementing variational inference in graphical models, though it is incremental as it focuses on deriving equations for a specific model.
The paper tackles the challenge of deriving variational message passing (VMP) update equations for latent Dirichlet allocation (LDA), a probabilistic model for topic discovery, by providing a detailed derivation that fills a gap in existing literature and software.
Latent Dirichlet Allocation (LDA) is a probabilistic model used to uncover latent topics in a corpus of documents. Inference is often performed using variational Bayes (VB) algorithms, which calculate a lower bound to the posterior distribution over the parameters. Deriving the variational update equations for new models requires considerable manual effort; variational message passing (VMP) has emerged as a "black-box" tool to expedite the process of variational inference. But applying VMP in practice still presents subtle challenges, and the existing literature does not contain the steps that are necessary to implement VMP for the standard smoothed LDA model, nor are available black-box probabilistic graphical modelling software able to do the word-topic updates necessary to implement LDA. In this paper, we therefore present a detailed derivation of the VMP update equations for LDA. We see this as a first step to enabling other researchers to calculate the VMP updates for similar graphical models.