On a continuous time model of gradient descent dynamics and instability in deep learning
This work addresses instability issues in gradient descent for deep learning practitioners, offering a theoretical tool and practical method, though it is incremental as it builds on existing optimization frameworks.
The authors tackled the problem of understanding gradient descent instability in deep learning by proposing the principal flow, a continuous-time model that captures divergent and oscillatory behaviors, and used this to develop a learning rate adaptation method that controls the trade-off between training stability and test performance.
The recipe behind the success of deep learning has been the combination of neural networks and gradient-based optimization. Understanding the behavior of gradient descent however, and particularly its instability, has lagged behind its empirical success. To add to the theoretical tools available to study gradient descent we propose the principal flow (PF), a continuous time flow that approximates gradient descent dynamics. To our knowledge, the PF is the only continuous flow that captures the divergent and oscillatory behaviors of gradient descent, including escaping local minima and saddle points. Through its dependence on the eigendecomposition of the Hessian the PF sheds light on the recently observed edge of stability phenomena in deep learning. Using our new understanding of instability we propose a learning rate adaptation method which enables us to control the trade-off between training stability and test set evaluation performance.