Analytic and Variational Stability in Deep Learning Systems
This work addresses the fundamental issue of stability for deep learning practitioners, offering a unified theoretical framework, but it is incremental as it builds on existing stability concepts.
The authors tackled the problem of stability in deep learning systems by proposing a unified analytic and variational framework, resulting in a theorem that links stability to Lyapunov-type energy dissipation and provides explicit stability exponents for various architectures and optimization methods.
We propose a unified analytic and variational framework for stability in deep learning systems viewed as coupled representation-parameter dynamics. The central object is the Learning Stability Profile, which measures how infinitesimal perturbations propagate through representations, parameters, and update mechanisms along the learning trajectory. Our main result, the Fundamental Analytic Stability Theorem, shows that uniform boundedness of these sensitivities is equivalent, up to norm equivalence, to the existence of a Lyapunov-type energy dissipating along the learning flow. In smooth regimes, this yields explicit stability exponents linking spectral norms, activation regularity, step sizes, and learning rates to contractive behavior. Classical spectral stability of feedforward networks, CFL-type conditions for residual architectures, and temporal stability laws for stochastic gradient methods follow as direct consequences. The framework extends to non-smooth systems, including ReLU networks, proximal and projected updates, and stochastic subgradient flows, by replacing classical derivatives with Clarke generalized derivatives and smooth energies with variational Lyapunov functionals. The resulting theory provides a unified dynamical description of stability across architectures and optimization methods, clarifying how design and training choices jointly control robustness and sensitivity to perturbations.