Directional Convergence, Benign Overfitting of Gradient Descent in leaky ReLU two-layer Neural Networks
This work addresses theoretical understanding of overfitting in neural networks for machine learning researchers, offering incremental improvements by extending directional convergence from gradient flow to gradient descent and broadening applicability.
The paper tackles the problem of benign overfitting in fixed-width leaky ReLU two-layer neural networks trained on mixture data via gradient descent, providing upper and lower classification error bounds and discovering a phase transition based on signal strength, with results that relax distributional assumptions like non-sub-Gaussian data and near orthogonality.
In this paper, we study benign overfitting of fixed width leaky ReLU two-layer neural network classifiers trained on mixture data via gradient descent. We provide both, upper and lower classification error bounds, and discover a phase transition in the bound as a function of signal strength. The lower bound leads to a characterization of cases when benign overfitting provably fails even if directional convergence occurs. Our analysis allows us to considerably relax the distributional assumptions that are made in existing work on benign overfitting of leaky ReLU two-layer neural network classifiers. We can allow for non-sub-Gaussian data and do not require near orthogonality. Our results are derived by establishing directional convergence of the network parameters and studying classification error bounds for the convergent direction. Previously, directional convergence in (leaky) ReLU neural networks was established only for gradient flow. By first establishing directional convergence, we are able to study benign overfitting of fixed width leaky ReLU two-layer neural network classifiers in a much wider range of scenarios than was done before.