Doubly Sparse Variational Gaussian Processes
This work addresses scalability issues for researchers and practitioners using Gaussian processes in machine learning, though it is incremental as it builds on existing methods.
The paper tackles the computational and memory limitations of Gaussian process models for large datasets by combining variational sparse approximations with state-space formulations, resulting in further computational and memory savings and enabling use in deep Gaussian process models.
The use of Gaussian process models is typically limited to datasets with a few tens of thousands of observations due to their complexity and memory footprint. The two most commonly used methods to overcome this limitation are 1) the variational sparse approximation which relies on inducing points and 2) the state-space equivalent formulation of Gaussian processes which can be seen as exploiting some sparsity in the precision matrix. We propose to take the best of both worlds: we show that the inducing point framework is still valid for state space models and that it can bring further computational and memory savings. Furthermore, we provide the natural gradient formulation for the proposed variational parameterisation. Finally, this work makes it possible to use the state-space formulation inside deep Gaussian process models as illustrated in one of the experiments.