Empirical Gaussian Processes
This addresses the problem of limited adaptivity and expert dependency in GP modeling for practitioners, representing a novel method for a known bottleneck rather than a foundational breakthrough.
The paper tackled the limitation of Gaussian processes (GPs) due to handcrafted kernel functions by introducing Empirical GPs, a framework that constructs data-driven priors by estimating mean and covariance functions from historical data, achieving competitive performance on learning curve extrapolation and time series forecasting benchmarks.
Gaussian processes (GPs) are powerful and widely used probabilistic regression models, but their effectiveness in practice is often limited by the choice of kernel function. This kernel function is typically handcrafted from a small set of standard functions, a process that requires expert knowledge, results in limited adaptivity to data, and imposes strong assumptions on the hypothesis space. We study Empirical GPs, a principled framework for constructing flexible, data-driven GP priors that overcome these limitations. Rather than relying on standard parametric kernels, we estimate the mean and covariance functions empirically from a corpus of historical observations, enabling the prior to reflect rich, non-trivial covariance structures present in the data. Theoretically, we show that the resulting model converges to the GP that is closest (in KL-divergence sense) to the real data generating process. Practically, we formulate the problem of learning the GP prior from independent datasets as likelihood estimation and derive an Expectation-Maximization algorithm with closed-form updates, allowing the model handle heterogeneous observation locations across datasets. We demonstrate that Empirical GPs achieve competitive performance on learning curve extrapolation and time series forecasting benchmarks.