Local Random Feature Approximations of the Gaussian Kernel
This work addresses the problem of poor kernel approximations for high-frequency data in machine learning, offering an incremental improvement over existing methods.
The authors tackled the scalability issue of Gaussian kernel models by proposing a localization scheme for random feature approximations, which significantly improved performance on high-frequency data, as demonstrated in Gaussian process regression experiments.
A fundamental drawback of kernel-based statistical models is their limited scalability to large data sets, which requires resorting to approximations. In this work, we focus on the popular Gaussian kernel and on techniques to linearize kernel-based models by means of random feature approximations. In particular, we do so by studying a less explored random feature approximation based on Maclaurin expansions and polynomial sketches. We show that such approaches yield poor results when modelling high-frequency data, and we propose a novel localization scheme that improves kernel approximations and downstream performance significantly in this regime. We demonstrate these gains on a number of experiments involving the application of Gaussian process regression to synthetic and real-world data of different data sizes and dimensions.