Near-Optimal Approximations for Bayesian Inference in Function Space
This work addresses scalability issues in Bayesian inference for machine learning and statistics, offering a non-parametric method that improves upon existing constrained approaches, though it is incremental in building on variational and diffusion-based techniques.
The paper tackles the problem of scalable Bayesian inference in function space by proposing an algorithm that approximates infinite-dimensional Langevin diffusion via projection, achieving near-optimal approximations with computational scaling of O(M^3+JM^2). It recovers sparse variational Gaussian processes as a special case and provides provable closeness to optimal variational approximations for convex and Lipschitz continuous likelihoods.
We propose a scalable inference algorithm for Bayes posteriors defined on a reproducing kernel Hilbert space (RKHS). Given a likelihood function and a Gaussian random element representing the prior, the corresponding Bayes posterior measure $Π_{\text{B}}$ can be obtained as the stationary distribution of an RKHS-valued Langevin diffusion. We approximate the infinite-dimensional Langevin diffusion via a projection onto the first $M$ components of the Kosambi-Karhunen-Loève expansion. Exploiting the thus obtained approximate posterior for these $M$ components, we perform inference for $Π_{\text{B}}$ by relying on the law of total probability and a sufficiency assumption. The resulting method scales as $O(M^3+JM^2)$, where $J$ is the number of samples produced from the posterior measure $Π_{\text{B}}$. Interestingly, the algorithm recovers the posterior arising from the sparse variational Gaussian process (SVGP) (see Titsias, 2009) as a special case, owed to the fact that the sufficiency assumption underlies both methods. However, whereas the SVGP is parametrically constrained to be a Gaussian process, our method is based on a non-parametric variational family $\mathcal{P}(\mathbb{R}^M)$ consisting of all probability measures on $\mathbb{R}^M$. As a result, our method is provably close to the optimal $M$-dimensional variational approximation of the Bayes posterior $Π_{\text{B}}$ in $\mathcal{P}(\mathbb{R}^M)$ for convex and Lipschitz continuous negative log likelihoods, and coincides with SVGP for the special case of a Gaussian error likelihood.