Augmented Message Passing Stein Variational Gradient Descent
This addresses a convergence limitation in a popular Bayesian inference method, offering an improvement for practitioners in fields like machine learning and statistics, though it appears incremental as it builds on existing SVGD approaches.
The paper tackles the variance collapse problem in Stein Variational Gradient Descent (SVGD), which reduces accuracy and diversity in Bayesian inference, by proposing Augmented Message Passing SVGD (AUMP-SVGD), a two-stage optimization method that overcomes this issue and achieves satisfactory accuracy in benchmark problems.
Stein Variational Gradient Descent (SVGD) is a popular particle-based method for Bayesian inference. However, its convergence suffers from the variance collapse, which reduces the accuracy and diversity of the estimation. In this paper, we study the isotropy property of finite particles during the convergence process and show that SVGD of finite particles cannot spread across the entire sample space. Instead, all particles tend to cluster around the particle center within a certain range and we provide an analytical bound for this cluster. To further improve the effectiveness of SVGD for high-dimensional problems, we propose the Augmented Message Passing SVGD (AUMP-SVGD) method, which is a two-stage optimization procedure that does not require sparsity of the target distribution, unlike the MP-SVGD method. Our algorithm achieves satisfactory accuracy and overcomes the variance collapse problem in various benchmark problems.