How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient Descent
This work provides refined theoretical insights for researchers in machine learning theory, focusing on shallow networks, but it is incremental as it builds upon prior NTK analyses.
The paper tackles the problem of determining the required number of neurons for generalization in two-layer neural networks trained with gradient descent in the NTK regime, deriving minimax optimal convergence rates and improving existing bounds on neuron count.
We analyze the generalization properties of two-layer neural networks in the neural tangent kernel (NTK) regime, trained with gradient descent (GD). For early stopped GD we derive fast rates of convergence that are known to be minimax optimal in the framework of non-parametric regression in reproducing kernel Hilbert spaces. On our way, we precisely keep track of the number of hidden neurons required for generalization and improve over existing results. We further show that the weights during training remain in a vicinity around initialization, the radius being dependent on structural assumptions such as degree of smoothness of the regression function and eigenvalue decay of the integral operator associated to the NTK.