A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks
This work addresses a theoretical gap in deep learning for researchers, but it is incremental as it extends existing NTK analysis to more realistic training conditions.
The paper tackles the mismatch between neural tangent kernel (NTK) theory and practice by analyzing noisy gradient descent with weight decay, showing it exhibits kernel-like behavior and achieves linear convergence of training loss up to a certain accuracy, with a novel generalization error bound for two-layer neural networks.
A recent breakthrough in deep learning theory shows that the training of over-parameterized deep neural networks can be characterized by a kernel function called \textit{neural tangent kernel} (NTK). However, it is known that this type of results does not perfectly match the practice, as NTK-based analysis requires the network weights to stay very close to their initialization throughout training, and cannot handle regularizers or gradient noises. In this paper, we provide a generalized neural tangent kernel analysis and show that noisy gradient descent with weight decay can still exhibit a "kernel-like" behavior. This implies that the training loss converges linearly up to a certain accuracy. We also establish a novel generalization error bound for two-layer neural networks trained by noisy gradient descent with weight decay.