Exact marginal inference in Latent Dirichlet Allocation
This provides an efficient solution for Bayesian inference in high-dimensional LDA applications, such as location-based data, though it is incremental as it builds on existing LDA frameworks.
The paper tackles exact marginal inference in Latent Dirichlet Allocation (LDA) for scenarios with many potential causes but few observations, showing that the exact Bayesian estimate can be computed in linear time in the number of causes with a simple formula, and generalizes this to sparse probabilities with limited tree width, while proving NP-hardness without such constraints.
Assume we have potential "causes" $z\in Z$, which produce "events" $w$ with known probabilities $β(w|z)$. We observe $w_1,w_2,...,w_n$, what can we say about the distribution of the causes? A Bayesian estimate will assume a prior on distributions on $Z$ (we assume a Dirichlet prior) and calculate a posterior. An average over that posterior then gives a distribution on $Z$, which estimates how much each cause $z$ contributed to our observations. This is the setting of Latent Dirichlet Allocation, which can be applied e.g. to topics "producing" words in a document. In this setting usually the number of observed words is large, but the number of potential topics is small. We are here interested in applications with many potential "causes" (e.g. locations on the globe), but only a few observations. We show that the exact Bayesian estimate can be computed in linear time (and constant space) in $|Z|$ for a given upper bound on $n$ with a surprisingly simple formula. We generalize this algorithm to the case of sparse probabilities $β(w|z)$, in which we only need to assume that the tree width of an "interaction graph" on the observations is limited. On the other hand we also show that without such limitation the problem is NP-hard.