The Renyi Gaussian Process: Towards Improved Generalization
This work addresses the need for better regularization and generalization in Gaussian process models, which are widely used in machine learning for uncertainty quantification, but it appears incremental as it builds on existing variational GP methods.
The paper tackles the problem of improving generalization in Gaussian process (GP) regression by introducing a new closed-form lower bound on the GP likelihood based on the Rényi α-divergence, which allows for tunable regularization and shows experimental improvements over existing GP inference methods.
We introduce an alternative closed form lower bound on the Gaussian process ($\mathcal{GP}$) likelihood based on the Rényi $α$-divergence. This new lower bound can be viewed as a convex combination of the Nyström approximation and the exact $\mathcal{GP}$. The key advantage of this bound, is its capability to control and tune the enforced regularization on the model and thus is a generalization of the traditional variational $\mathcal{GP}$ regression. From a theoretical perspective, we provide the convergence rate and risk bound for inference using our proposed approach. Experiments on real data show that the proposed algorithm may be able to deliver improvement over several $\mathcal{GP}$ inference methods.