Towards Improved Learning in Gaussian Processes: The Best of Two Worlds
This work addresses a specific issue in Gaussian process training for practitioners, but it is incremental as it builds on existing methods like EP and VI.
The paper tackled the problem of hyperparameter optimization in Gaussian processes with non-Gaussian likelihoods by designing a hybrid training procedure that combines conjugate-computation variational inference for posterior inference with an Expectation Propagation-like marginal likelihood approximation for learning. The result showed improved generalization in binary classification tasks.
Gaussian process training decomposes into inference of the (approximate) posterior and learning of the hyperparameters. For non-Gaussian (non-conjugate) likelihoods, two common choices for approximate inference are Expectation Propagation (EP) and Variational Inference (VI), which have complementary strengths and weaknesses. While VI's lower bound to the marginal likelihood is a suitable objective for inferring the approximate posterior, it does not automatically imply it is a good learning objective for hyperparameter optimization. We design a hybrid training procedure where the inference leverages conjugate-computation VI and the learning uses an EP-like marginal likelihood approximation. We empirically demonstrate on binary classification that this provides a good learning objective and generalizes better.