Gradient Descent Finds Over-Parameterized Neural Networks with Sharp Generalization for Nonparametric Regression
This work addresses the problem of understanding generalization in over-parameterized neural networks for nonparametric regression, providing theoretical guarantees that bridge the gap between neural networks and kernel methods, which is incremental but clarifies open questions in the literature.
The paper tackles nonparametric regression using over-parameterized two-layer neural networks trained by gradient descent with early stopping, showing that the trained network achieves a sharp risk rate of O(ε_n^2), matching the rate of classical kernel regression and being minimax optimal in specific cases, without requiring distributional assumptions on bounded covariates.
We study nonparametric regression by an over-parameterized two-layer neural network trained by gradient descent (GD) in this paper. We show that, if the neural network is trained by GD with early stopping, then the trained network renders a sharp rate of the nonparametric regression risk of $\mathcal{O}(ε_n^2)$, which is the same rate as that for the classical kernel regression trained by GD with early stopping, where $ε_n$ is the critical population rate of the Neural Tangent Kernel (NTK) associated with the network and $n$ is the size of the training data. It is remarked that our result does not require distributional assumptions about the covariate as long as the covariate is bounded, in a strong contrast with many existing results which rely on specific distributions of the covariates such as the spherical uniform data distribution or distributions satisfying certain restrictive conditions. The rate $\mathcal{O}(ε_n^2)$ is known to be minimax optimal for specific cases, such as the case that the NTK has a polynomial eigenvalue decay rate which happens under certain distributional assumptions on the covariates. Our result formally fills the gap between training a classical kernel regression model and training an over-parameterized but finite-width neural network by GD for nonparametric regression without distributional assumptions on the bounded covariate. We also provide confirmative answers to certain open questions or address particular concerns in the literature of training over-parameterized neural networks by GD with early stopping for nonparametric regression, including the characterization of the stopping time, the lower bound for the network width, and the constant learning rate used in GD.