Multirate Stein Variational Gradient Descent for Efficient Bayesian Sampling
This work addresses a bottleneck in Bayesian inference for researchers and practitioners dealing with high-dimensional or complex posteriors, offering incremental improvements to SVGD.
The paper tackled the problem of inefficient and unstable Bayesian sampling in Stein variational gradient descent (SVGD) by developing multirate versions that update attraction and repulsion components at different time scales, resulting in improved robustness and quality-cost tradeoffs across six benchmark families, with adaptive multirate SVGD showing the strongest gains on stiff hierarchical, anisotropic, and multimodal targets.
Many particle-based Bayesian inference methods use a single global step size for all parts of the update. In Stein variational gradient descent (SVGD), however, each update combines two qualitatively different effects: attraction toward high-posterior regions and repulsion that preserves particle diversity. These effects can evolve at different rates, especially in high-dimensional, anisotropic, or hierarchical posteriors, so one step size can be unstable in some regions and inefficient in others. We derive a multirate version of SVGD that updates these components on different time scales. The framework yields practical algorithms, including a symmetric split method, a fixed multirate method (MR-SVGD), and an adaptive multirate method (Adapt-MR-SVGD) with local error control. We evaluate the methods in a broad and rigorous benchmark suite covering six problem families: a 50D Gaussian target, multiple 2D synthetic targets, UCI Bayesian logistic regression, multimodal Gaussian mixtures, Bayesian neural networks, and large-scale hierarchical logistic regression. Evaluation includes posterior-matching metrics, predictive performance, calibration quality, mixing, and explicit computational cost accounting. Across these six benchmark families, multirate SVGD variants improve robustness and quality-cost tradeoffs relative to vanilla SVGD. The strongest gains appear on stiff hierarchical, strongly anisotropic, and multimodal targets, where adaptive multirate SVGD is usually the strongest variant and fixed multirate SVGD provides a simpler robust alternative at lower cost.