Rates of Convergence for Sparse Variational Gaussian Process Regression
This provides a theoretical foundation for efficient scaling of Gaussian processes in large datasets, particularly useful for continual learning scenarios.
The paper tackles the computational scaling of variational Gaussian process regression by analyzing how the number of inducing variables must grow to maintain approximation quality, showing that for regression with normally distributed inputs and a Squared Exponential kernel, M = O(log^D N) is sufficient to make the KL divergence arbitrarily small.
Excellent variational approximations to Gaussian process posteriors have been developed which avoid the $\mathcal{O}\left(N^3\right)$ scaling with dataset size $N$. They reduce the computational cost to $\mathcal{O}\left(NM^2\right)$, with $M\ll N$ being the number of inducing variables, which summarise the process. While the computational cost seems to be linear in $N$, the true complexity of the algorithm depends on how $M$ must increase to ensure a certain quality of approximation. We address this by characterising the behavior of an upper bound on the KL divergence to the posterior. We show that with high probability the KL divergence can be made arbitrarily small by growing $M$ more slowly than $N$. A particular case of interest is that for regression with normally distributed inputs in D-dimensions with the popular Squared Exponential kernel, $M=\mathcal{O}(\log^D N)$ is sufficient. Our results show that as datasets grow, Gaussian process posteriors can truly be approximated cheaply, and provide a concrete rule for how to increase $M$ in continual learning scenarios.