Agnostic Learning of General ReLU Activation Using Gradient Descent
This addresses a theoretical gap in neural network optimization for practitioners by extending analysis to non-zero bias, though it is incremental as it builds on prior zero-bias work.
The paper tackles the problem of agnostically learning a single ReLU function with non-zero bias under Gaussian distributions, showing that gradient descent from random initialization achieves an error within a constant factor of the optimal error in polynomial iterations.
We provide a convergence analysis of gradient descent for the problem of agnostically learning a single ReLU function with moderate bias under Gaussian distributions. Unlike prior work that studies the setting of zero bias, we consider the more challenging scenario when the bias of the ReLU function is non-zero. Our main result establishes that starting from random initialization, in a polynomial number of iterations gradient descent outputs, with high probability, a ReLU function that achieves an error that is within a constant factor of the optimal error of the best ReLU function with moderate bias. We also provide finite sample guarantees, and these techniques generalize to a broader class of marginal distributions beyond Gaussians.