To discretize continually: Mean shift interacting particle systems for Bayesian inference
Provides a new paradigm for Bayesian inference by enabling deterministic quadrature via particle systems that avoid mode collapse and scale to high dimensions, addressing limitations of MCMC and variational inference.
The paper introduces interacting particle systems that minimize maximum mean discrepancy to approximate expectations under unnormalized densities, extending mean shift to continuous distributions. The methods converge quickly, handle anisotropy and multi-modality, and scale to high dimensions, outperforming baselines on benchmarks including Bayesian hierarchical models and PDE-constrained inverse problems.
Integration against a probability distribution given its unnormalized density is a central task in Bayesian inference and other fields. We introduce new methods for approximating such expectations with a small set of weighted samples -- i.e., a quadrature rule -- constructed via an interacting particle system that minimizes maximum mean discrepancy (MMD) to the target distribution. These methods extend the classical mean shift algorithm, as well as recent algorithms for optimal quantization of empirical distributions, to the case of continuous distributions. Crucially, our approach creates dynamics for MMD minimization that are invariant to the unknown normalizing constant; they also admit both gradient-free and gradient-informed implementations. The resulting mean shift interacting particle systems converge quickly, capture anisotropy and multi-modality, avoid mode collapse, and scale to high dimensions. We demonstrate their performance on a wide range of benchmark sampling problems, including multi-modal mixtures, Bayesian hierarchical models, PDE-constrained inverse problems, and beyond.