Recursive Sampling for the Nyström Method
This addresses the scalability bottleneck in kernel methods for machine learning practitioners, offering a significant improvement over prior techniques.
The paper tackles the problem of kernel Nyström approximation by developing an algorithm that runs in linear time relative to training points and is provably accurate for all kernel matrices, achieving more accurate, lower rank approximations faster than existing methods like uniformly sampled Nyström and random Fourier features.
We give the first algorithm for kernel Nyström approximation that runs in *linear time in the number of training points* and is provably accurate for all kernel matrices, without dependence on regularity or incoherence conditions. The algorithm projects the kernel onto a set of $s$ landmark points sampled by their *ridge leverage scores*, requiring just $O(ns)$ kernel evaluations and $O(ns^2)$ additional runtime. While leverage score sampling has long been known to give strong theoretical guarantees for Nyström approximation, by employing a fast recursive sampling scheme, our algorithm is the first to make the approach scalable. Empirically we show that it finds more accurate, lower rank kernel approximations in less time than popular techniques such as uniformly sampled Nyström approximation and the random Fourier features method.