Approximate Inference for Fully Bayesian Gaussian Process Regression
This work addresses hyperparameter uncertainty in Gaussian Process Regression for machine learning practitioners, but it is incremental as it applies existing approximation methods to a known hierarchical specification.
The paper tackled the problem of learning hyperparameters in Gaussian Process Regression by proposing fully Bayesian inference with two approximation schemes, HMC and VI, and found that these methods improved predictive performance on benchmark datasets.
Learning in Gaussian Process models occurs through the adaptation of hyperparameters of the mean and the covariance function. The classical approach entails maximizing the marginal likelihood yielding fixed point estimates (an approach called \textit{Type II maximum likelihood} or ML-II). An alternative learning procedure is to infer the posterior over hyperparameters in a hierarchical specification of GPs we call \textit{Fully Bayesian Gaussian Process Regression} (GPR). This work considers two approximation schemes for the intractable hyperparameter posterior: 1) Hamiltonian Monte Carlo (HMC) yielding a sampling-based approximation and 2) Variational Inference (VI) where the posterior over hyperparameters is approximated by a factorized Gaussian (mean-field) or a full-rank Gaussian accounting for correlations between hyperparameters. We analyze the predictive performance for fully Bayesian GPR on a range of benchmark data sets.