Gradient Descent in Neural Networks as Sequential Learning in RKBS
This provides a more general theoretical framework for analyzing neural network training across all widths, addressing a foundational problem in machine learning theory.
The paper tackles the limitation of Neural Tangent Kernels (NTKs) in approximating neural network training, especially for narrower networks, by developing an exact representation using reproducing kernel Banach spaces (RKBS). It proves that gradient descent training can be exactly replicated by sequential learning in RKBS, leading to new bounds on uniform convergence that incorporate iteration count and learning rate.
The study of Neural Tangent Kernels (NTKs) has provided much needed insight into convergence and generalization properties of neural networks in the over-parametrized (wide) limit by approximating the network using a first-order Taylor expansion with respect to its weights in the neighborhood of their initialization values. This allows neural network training to be analyzed from the perspective of reproducing kernel Hilbert spaces (RKHS), which is informative in the over-parametrized regime, but a poor approximation for narrower networks as the weights change more during training. Our goal is to extend beyond the limits of NTK toward a more general theory. We construct an exact power-series representation of the neural network in a finite neighborhood of the initial weights as an inner product of two feature maps, respectively from data and weight-step space, to feature space, allowing neural network training to be analyzed from the perspective of reproducing kernel {\em Banach} space (RKBS). We prove that, regardless of width, the training sequence produced by gradient descent can be exactly replicated by regularized sequential learning in RKBS. Using this, we present novel bound on uniform convergence where the iterations count and learning rate play a central role, giving new theoretical insight into neural network training.