Feature Learning Beyond the Edge of Stability
This addresses the challenge of training stability and feature quality in deep learning, though it appears incremental as it builds on existing edge-of-stability research.
The paper tackles the problem of training deep neural networks beyond the edge of stability by proposing a homogeneous multilayer perceptron with specific width patterns and gradient scaling, resulting in improved feature learning and implicit sharpness regularization without numerical issues.
We propose a homogeneous multilayer perceptron parameterization with polynomial hidden layer width pattern and analyze its training dynamics under stochastic gradient descent with depthwise gradient scaling in a general supervised learning scenario. We obtain formulas for the first three Taylor coefficients of the minibatch loss during training that illuminate the connection between sharpness and feature learning, providing in particular a soft rank variant that quantifies the quality of learned hidden layer features. Based on our theory, we design a gradient scaling scheme that in tandem with a quadratic width pattern enables training beyond the edge of stability without loss explosions or numerical errors, resulting in improved feature learning and implicit sharpness regularization as demonstrated empirically.