Unbiased Stochastic Optimization for Gaussian Processes on Finite Dimensional RKHS
This work addresses a specific bottleneck in Gaussian Process optimization for researchers and practitioners dealing with limited computational resources.
The paper tackles the problem of biased stochastic gradient approximations in Gaussian Process hyperparameter learning by proposing exact stochastic inference algorithms for kernels with finite-dimensional RKHS, achieving better experimental results under memory constraints.
Current methods for stochastic hyperparameter learning in Gaussian Processes (GPs) rely on approximations, such as computing biased stochastic gradients or using inducing points in stochastic variational inference. However, when using such methods we are not guaranteed to converge to a stationary point of the true marginal likelihood. In this work, we propose algorithms for exact stochastic inference of GPs with kernels that induce a Reproducing Kernel Hilbert Space (RKHS) of moderate finite dimension. Our approach can also be extended to infinite dimensional RKHSs at the cost of forgoing exactness. Both for finite and infinite dimensional RKHSs, our method achieves better experimental results than existing methods when memory resources limit the feasible batch size and the possible number of inducing points.