Gradient Descent as a Perceptron Algorithm: Understanding Dynamics and Implicit Acceleration
This provides a new perspective on optimization dynamics for neural networks, potentially advancing research in the field, though it is incremental in building on classical perceptron algorithms.
The paper tackles the challenge of understanding gradient descent dynamics in neural network training by showing that its steps reduce to those of generalized perceptron algorithms, and demonstrates that nonlinearity in a two-layer model yields a faster iteration complexity of $ ilde{O}(\sqrt{d})$ compared to $Ω(d)$ for linear models.
Even for the gradient descent (GD) method applied to neural network training, understanding its optimization dynamics, including convergence rate, iterate trajectories, function value oscillations, and especially its implicit acceleration, remains a challenging problem. We analyze nonlinear models with the logistic loss and show that the steps of GD reduce to those of generalized perceptron algorithms (Rosenblatt, 1958), providing a new perspective on the dynamics. This reduction yields significantly simpler algorithmic steps, which we analyze using classical linear algebra tools. Using these tools, we demonstrate on a minimalistic example that the nonlinearity in a two-layer model can provably yield a faster iteration complexity $\tilde{O}(\sqrt{d})$ compared to $Ω(d)$ achieved by linear models, where $d$ is the number of features. This helps explain the optimization dynamics and the implicit acceleration phenomenon observed in neural networks. The theoretical results are supported by extensive numerical experiments. We believe that this alternative view will further advance research on the optimization of neural networks.