Variational Orthogonal Features
This work addresses computational bottlenecks for researchers and practitioners applying Gaussian process models to large datasets, representing an incremental improvement over prior methods.
The paper tackles the computational cost of sparse stochastic variational inference for Gaussian processes by introducing variational orthogonal features that reduce the per-iteration cost from O(˜NM^2 + M^3) to O(˜NT + M^2T) or O(˜NT + MT) with an approximation, and analyzes the impact of this approximation on inference quality.
Sparse stochastic variational inference allows Gaussian process models to be applied to large datasets. The per iteration computational cost of inference with this method is $\mathcal{O}(\tilde{N}M^2+M^3),$ where $\tilde{N}$ is the number of points in a minibatch and $M$ is the number of `inducing features', which determine the expressiveness of the variational family. Several recent works have shown that for certain priors, features can be defined that remove the $\mathcal{O}(M^3)$ cost of computing a minibatch estimate of an evidence lower bound (ELBO). This represents a significant computational savings when $M\gg \tilde{N}$. We present a construction of features for any stationary prior kernel that allow for computation of an unbiased estimator to the ELBO using $T$ Monte Carlo samples in $\mathcal{O}(\tilde{N}T+M^2T)$ and in $\mathcal{O}(\tilde{N}T+MT)$ with an additional approximation. We analyze the impact of this additional approximation on inference quality.