Knot Selection in Sparse Gaussian Processes
This addresses a scalability issue for practitioners using sparse Gaussian process approximations, though it is an incremental improvement on existing knot selection techniques.
The paper tackles the problem of suboptimal knot selection in sparse Gaussian processes, which leads to unnecessary computational cost without accuracy gains, by proposing a one-at-a-time algorithm using Bayesian optimization that improves both accuracy and speed over standard methods.
Knot-based, sparse Gaussian processes have enjoyed considerable success as scalable approximations to full Gaussian processes. Problems can occur, however, when knot selection is done by optimizing the marginal likelihood. For example, the marginal likelihood surface is highly multimodal, which can cause suboptimal knot placement where some knots serve practically no function. This is especially a problem when many more knots are used than are necessary, resulting in extra computational cost for little to no gains in accuracy. We propose a one-at-a-time knot selection algorithm to select both the number and placement of knots. Our algorithm uses Bayesian optimization to efficiently propose knots that are likely to be good and largely avoids the pathologies encountered when using the marginal likelihood as the objective function. We provide empirical results showing improved accuracy and speed over the current standard approaches.