Revisiting Active Sets for Gaussian Process Decoders
This work addresses scalability and training issues in Gaussian process decoders for machine learning practitioners, representing an incremental improvement over existing methods.
The paper tackled the high computational cost and training difficulty of Gaussian process decoders by developing a stochastic active sets approximation, which improved robustness and reduced computational expense while achieving better latent space structure and representation learning than variational autoencoders.
Decoders built on Gaussian processes (GPs) are enticing due to the marginalisation over the non-linear function space. Such models (also known as GP-LVMs) are often expensive and notoriously difficult to train in practice, but can be scaled using variational inference and inducing points. In this paper, we revisit active set approximations. We develop a new stochastic estimate of the log-marginal likelihood based on recently discovered links to cross-validation, and propose a computationally efficient approximation thereof. We demonstrate that the resulting stochastic active sets (SAS) approximation significantly improves the robustness of GP decoder training while reducing computational cost. The SAS-GP obtains more structure in the latent space, scales to many datapoints and learns better representations than variational autoencoders, which is rarely the case for GP decoders.