When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?
This work provides theoretical guarantees for the convergence of gradient descent in deep learning, which is important for researchers and practitioners using smoothed ReLU activations.
This paper investigates the conditions under which gradient descent, when applied to deep networks with smoothed ReLU activations and logistic loss, can achieve zero loss. The authors establish two sufficient conditions for this interpolation: a bound on the initial loss or a data separation condition.
We establish conditions under which gradient descent applied to fixed-width deep networks drives the logistic loss to zero, and prove bounds on the rate of convergence. Our analysis applies for smoothed approximations to the ReLU, such as Swish and the Huberized ReLU, proposed in previous applied work. We provide two sufficient conditions for convergence. The first is simply a bound on the loss at initialization. The second is a data separation condition used in prior analyses.