Automatic Variational Inference in Stan
This enables scalable approximate Bayesian inference for users of probabilistic programming frameworks like Stan, reducing manual effort and expanding model support.
The paper tackles the problem of automating variational inference for Bayesian models, which traditionally requires manual derivations, by proposing automatic differentiation variational inference (ADVI) that works with any model in Stan, achieving scalability and broad applicability without conjugacy assumptions.
Variational inference is a scalable technique for approximate Bayesian inference. Deriving variational inference algorithms requires tedious model-specific calculations; this makes it difficult to automate. We propose an automatic variational inference algorithm, automatic differentiation variational inference (ADVI). The user only provides a Bayesian model and a dataset; nothing else. We make no conjugacy assumptions and support a broad class of models. The algorithm automatically determines an appropriate variational family and optimizes the variational objective. We implement ADVI in Stan (code available now), a probabilistic programming framework. We compare ADVI to MCMC sampling across hierarchical generalized linear models, nonconjugate matrix factorization, and a mixture model. We train the mixture model on a quarter million images. With ADVI we can use variational inference on any model we write in Stan.