A Simple Algorithm For Scaling Up Kernel Methods
This addresses the scalability problem for researchers and practitioners using kernel methods in machine learning, though it appears incremental as it builds on existing random feature techniques.
The paper tackles the computational and memory limitations of kernel methods for large datasets by introducing a novel random feature regression algorithm that scales to virtually infinite random features, demonstrating performance on CIFAR-10.
The recent discovery of the equivalence between infinitely wide neural networks (NNs) in the lazy training regime and Neural Tangent Kernels (NTKs) (Jacot et al., 2018) has revived interest in kernel methods. However, conventional wisdom suggests kernel methods are unsuitable for large samples due to their computational complexity and memory requirements. We introduce a novel random feature regression algorithm that allows us (when necessary) to scale to virtually infinite numbers of random features. We illustrate the performance of our method on the CIFAR-10 dataset.