Knot Selection in Sparse Gaussian Processes with a Variational Objective Function
This work addresses scalability issues in Gaussian process approximations for machine learning practitioners, but it is incremental as it builds on existing variational methods.
The paper tackles the slow optimization and lack of methods for selecting the number of knots in sparse Gaussian processes by proposing a one-at-a-time knot selection algorithm using Bayesian optimization, achieving competitive performance on three benchmark datasets at a fraction of the computational cost.
Sparse, knot-based Gaussian processes have enjoyed considerable success as scalable approximations to full Gaussian processes. Certain sparse models can be derived through specific variational approximations to the true posterior, and knots can be selected to minimize the Kullback-Leibler divergence between the approximate and true posterior. While this has been a successful approach, simultaneous optimization of knots can be slow due to the number of parameters being optimized. Furthermore, there have been few proposed methods for selecting the number of knots, and no experimental results exist in the literature. We propose using a one-at-a-time knot selection algorithm based on Bayesian optimization to select the number and locations of knots. We showcase the competitive performance of this method relative to simultaneous optimization of knots on three benchmark data sets, but at a fraction of the computational cost.