Variable noise and dimensionality reduction for sparse Gaussian processes
This work addresses scalability issues for Gaussian processes in machine learning, making them more practical for high-dimensional and noisy real-world datasets, though it is incremental as it builds on existing SPGP methods.
The paper tackles the computational and scalability limitations of sparse pseudo-input Gaussian processes (SPGP) in high-dimensional data by introducing automatic dimensionality reduction and input-dependent noise modeling, enabling application to larger and more complex datasets with reduced complexity from N^3 to NM^2.
The sparse pseudo-input Gaussian process (SPGP) is a new approximation method for speeding up GP regression in the case of a large number of data points N. The approximation is controlled by the gradient optimization of a small set of M `pseudo-inputs', thereby reducing complexity from N^3 to NM^2. One limitation of the SPGP is that this optimization space becomes impractically big for high dimensional data sets. This paper addresses this limitation by performing automatic dimensionality reduction. A projection of the input space to a low dimensional space is learned in a supervised manner, alongside the pseudo-inputs, which now live in this reduced space. The paper also investigates the suitability of the SPGP for modeling data with input-dependent noise. A further extension of the model is made to make it even more powerful in this regard - we learn an uncertainty parameter for each pseudo-input. The combination of sparsity, reduced dimension, and input-dependent noise makes it possible to apply GPs to much larger and more complex data sets than was previously practical. We demonstrate the benefits of these methods on several synthetic and real world problems.