Random feature approximation for general spectral methods
This work provides a theoretical foundation for analyzing neural networks and neural operators via the Neural Tangent Kernel approach, addressing a gap in understanding for large-scale kernel methods.
The authors extended the theoretical analysis of random feature methods from Tikhonov regularization to a broad class of spectral regularization techniques, including gradient descent and accelerated algorithms, and obtained optimal learning rates over regularity classes, even for those not in the reproducing kernel Hilbert space.
Random feature approximation is arguably one of the most widely used techniques for kernel methods in large-scale learning algorithms. In this work, we analyze the generalization properties of random feature methods, extending previous results for Tikhonov regularization to a broad class of spectral regularization techniques. This includes not only explicit methods but also implicit schemes such as gradient descent and accelerated algorithms like the Heavy-Ball and Nesterov method. Through this framework, we enable a theoretical analysis of neural networks and neural operators through the lens of the Neural Tangent Kernel (NTK) approach trained via gradient descent. For our estimators we obtain optimal learning rates over regularity classes (even for classes that are not included in the reproducing kernel Hilbert space), which are defined through appropriate source conditions. This improves or completes previous results obtained in related settings for specific kernel algorithms.