Online, Informative MCMC Thinning with Kernelized Stein Discrepancy
This addresses the problem of computational efficiency in Bayesian inference for practitioners, though it appears incremental as a variant of existing MCMC approaches.
The paper tackles the challenge of efficiently representing Bayesian posterior distributions by proposing KSD Thinning, an MCMC variant that retains only samples exceeding a Kernelized Stein Discrepancy threshold. It demonstrates superior consistency/complexity tradeoffs compared to other online nonparametric Bayesian methods in experiments.
A fundamental challenge in Bayesian inference is efficient representation of a target distribution. Many non-parametric approaches do so by sampling a large number of points using variants of Markov Chain Monte Carlo (MCMC). We propose an MCMC variant that retains only those posterior samples which exceed a KSD threshold, which we call KSD Thinning. We establish the convergence and complexity tradeoffs for several settings of KSD Thinning as a function of the KSD threshold parameter, sample size, and other problem parameters. Finally, we provide experimental comparisons against other online nonparametric Bayesian methods that generate low-complexity posterior representations, and observe superior consistency/complexity tradeoffs. Code is available at github.com/colehawkins/KSD-Thinning.