Accurate and Scalable Stochastic Gaussian Process Regression via Learnable Coreset-based Variational Inference
This work addresses scalability issues in Gaussian process regression for machine learning practitioners, offering an incremental improvement over existing stochastic variational methods.
The paper tackles the problem of scaling Gaussian process regression by introducing a stochastic variational inference method based on learnable coresets, achieving linear O(M) complexity in variational parameters while maintaining O(M^3) time and O(M^2) space complexity for numerical stability.
We introduce a novel stochastic variational inference method for Gaussian process ($\mathcal{GP}$) regression, by deriving a posterior over a learnable set of coresets: i.e., over pseudo-input/output, weighted pairs. Unlike former free-form variational families for stochastic inference, our coreset-based variational $\mathcal{GP}$ (CVGP) is defined in terms of the $\mathcal{GP}$ prior and the (weighted) data likelihood. This formulation naturally incorporates inductive biases of the prior, and ensures its kernel and likelihood dependencies are shared with the posterior. We derive a variational lower-bound on the log-marginal likelihood by marginalizing over the latent $\mathcal{GP}$ coreset variables, and show that CVGP's lower-bound is amenable to stochastic optimization. CVGP reduces the dimensionality of the variational parameter search space to linear $\mathcal{O}(M)$ complexity, while ensuring numerical stability at $\mathcal{O}(M^3)$ time complexity and $\mathcal{O}(M^2)$ space complexity.