A Theory of Saddle Escape in Deep Nonlinear Networks
For researchers studying training dynamics in deep neural networks, this work provides a theoretical understanding of saddle escape times in deep nonlinear networks, extending prior analyses from shallow and linear networks.
The paper derives an exact identity for the imbalance of Frobenius norms of layer weight matrices in deep nonlinear networks, classifying activation functions into four universality classes. It reduces the matrix flow to a scalar ODE, yielding a critical-depth escape time law τ_* = Θ(ε^{-(r-2)}) governed by the number r of bottleneck layers, with close agreement between theory and simulations.
In deep networks with small initialization, training exhibits long plateaus separated by sharp feature-acquisition transitions. Whereas shallow nonlinear networks and deep linear networks are well studied, extending these analyses to deep nonlinear networks remains challenging. We derive an exact identity for the imbalance of Frobenius norms of layer weight matrices that holds for any smooth activation and any differentiable loss and use this to classify activation functions into four universality classes. On the permutation-symmetric submanifold, the identity combines with an approximate balance law to reduce the full matrix flow to a scalar ODE, giving a critical-depth escape time law $τ_\star = Θ(\varepsilon^{-(r-2)})$ governed by the number $r$ of layers at the bottleneck scale rather than the total depth $L$. We find that this same $r-2$ exponent is recovered under He-normal initialization with $r$ bottleneck layers rescaled by $\varepsilon$, where the symmetry manifold is preserved by the flow but not attracting. We find close agreement between our theory and numerical simulations.