A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
Provides theoretical guarantees for neural network training dynamics and generalization, which is foundational for understanding optimization in deep learning.
The paper analyzes gradient descent dynamics for training two-layer neural networks with both layers updated, showing that in over-parametrized regimes it achieves zero training loss exponentially fast regardless of label quality, and establishes sharp generalization error estimates for target functions in reproducing kernel Hilbert spaces.
A fairly comprehensive analysis is presented for the gradient descent dynamics for training two-layer neural network models in the situation when the parameters in both layers are updated. General initialization schemes as well as general regimes for the network width and training data size are considered. In the over-parametrized regime, it is shown that gradient descent dynamics can achieve zero training loss exponentially fast regardless of the quality of the labels. In addition, it is proved that throughout the training process the functions represented by the neural network model are uniformly close to that of a kernel method. For general values of the network width and training data size, sharp estimates of the generalization error is established for target functions in the appropriate reproducing kernel Hilbert space.