Recurrent Memory for Online Interdomain Gaussian Processes
This work addresses the challenge of efficient online learning with long-term dependencies for applications like time series prediction and continual learning, representing an incremental advancement by adapting HiPPO from RNNs to Gaussian processes.
The authors tackled the problem of capturing long-term memory in sequential data for online learning by proposing OHSVGP, a novel online Gaussian process model that integrates HiPPO projections as inducing variables, resulting in improved predictive performance, long-term memory preservation, and computational efficiency compared to existing methods.
We propose a novel online Gaussian process (GP) model that is capable of capturing long-term memory in sequential data in an online learning setting. Our model, Online HiPPO Sparse Variational Gaussian Process (OHSVGP), leverages the HiPPO (High-order Polynomial Projection Operators) framework, which is popularized in the RNN domain due to its long-range memory modeling capabilities. We interpret the HiPPO time-varying orthogonal projections as inducing variables with time-dependent orthogonal polynomial basis functions, which allows the SVGP inducing variables to memorize the process history. We show that the HiPPO framework fits naturally into the interdomain GP framework and demonstrate that the kernel matrices can also be updated online in a recurrence form based on the ODE evolution of HiPPO. We evaluate OHSVGP with online prediction for 1D time series, continual learning in discriminative GP model for data with multidimensional inputs, and deep generative modeling with sparse Gaussian process variational autoencoder, showing that it outperforms existing online GP methods in terms of predictive performance, long-term memory preservation, and computational efficiency.