Semi-Implicit Variational Inference via Score Matching
This is an incremental improvement for researchers in Bayesian machine learning, addressing computational bottlenecks in variational inference.
The paper tackles the challenge of intractable densities in semi-implicit variational inference by proposing SIVI-SM, a method using score matching, which matches MCMC accuracy and outperforms ELBO-based methods in Bayesian inference tasks.
Semi-implicit variational inference (SIVI) greatly enriches the expressiveness of variational families by considering implicit variational distributions defined in a hierarchical manner. However, due to the intractable densities of variational distributions, current SIVI approaches often use surrogate evidence lower bounds (ELBOs) or employ expensive inner-loop MCMC runs for unbiased ELBOs for training. In this paper, we propose SIVI-SM, a new method for SIVI based on an alternative training objective via score matching. Leveraging the hierarchical structure of semi-implicit variational families, the score matching objective allows a minimax formulation where the intractable variational densities can be naturally handled with denoising score matching. We show that SIVI-SM closely matches the accuracy of MCMC and outperforms ELBO-based SIVI methods in a variety of Bayesian inference tasks.