Scalable Gaussian Processes with Low-Rank Deep Kernel Decomposition
This work addresses the problem of high inference costs and limited kernel flexibility in Gaussian processes for researchers and practitioners in machine learning, representing a novel method for a known bottleneck.
The paper tackled the challenge of designing expressive and scalable kernels for Gaussian processes by introducing a low-rank deep kernel representation that enables exact GP inference in linear time and memory without inducing points, achieving improved predictive accuracy, uncertainty quantification, and computational efficiency in experiments.
Kernels are key to encoding prior beliefs and data structures in Gaussian process (GP) models. The design of expressive and scalable kernels has garnered significant research attention. Deep kernel learning enhances kernel flexibility by feeding inputs through a neural network before applying a standard parametric form. However, this approach remains limited by the choice of base kernels, inherits high inference costs, and often demands sparse approximations. Drawing on Mercer's theorem, we introduce a fully data-driven, scalable deep kernel representation where a neural network directly represents a low-rank kernel through a small set of basis functions. This construction enables highly efficient exact GP inference in linear time and memory without invoking inducing points. It also supports scalable mini-batch training based on a principled variational inference framework. We further propose a simple variance correction procedure to guard against overconfidence in uncertainty estimates. Experiments on synthetic and real-world data demonstrate the advantages of our deep kernel GP in terms of predictive accuracy, uncertainty quantification, and computational efficiency.