Stochastic gradient variational Bayes for gamma approximating distributions
This work provides a method for scalable Bayesian inference in models with gamma priors, which is incremental as it extends existing gradient control techniques to a specific distribution type.
The paper tackled the challenge of controlling gradient variance in variational inference for gamma-distributed latent variables, enabling black-box inference for models requiring sparsity and non-negativity, and demonstrated improved performance over generic sampling and Gaussian-based methods on network data and sparse factor analysis.
While stochastic variational inference is relatively well known for scaling inference in Bayesian probabilistic models, related methods also offer ways to circumnavigate the approximation of analytically intractable expectations. The key challenge in either setting is controlling the variance of gradient estimates: recent work has shown that for continuous latent variables, particularly multivariate Gaussians, this can be achieved by using the gradient of the log posterior. In this paper we apply the same idea to gamma distributed latent variables given gamma variational distributions, enabling straightforward "black box" variational inference in models where sparsity and non-negativity are appropriate. We demonstrate the method on a recently proposed gamma process model for network data, as well as a novel sparse factor analysis. We outperform generic sampling algorithms and the approach of using Gaussian variational distributions on transformed variables.