Stability of Deep Neural Networks via discrete rough paths
This work addresses stability analysis for deep learning practitioners, offering a novel mathematical framework that is incremental in applying rough path theory to neural networks.
The paper tackles the problem of analyzing the stability of deep residual neural networks by providing a priori estimates for network outputs in terms of input data and trained weights, using rough path theory to handle rough weight functions and showing bounded estimates even for weights behaving like Brownian motions.
Using rough path techniques, we provide a priori estimates for the output of Deep Residual Neural Networks in terms of both the input data and the (trained) network weights. As trained network weights are typically very rough when seen as functions of the layer, we propose to derive stability bounds in terms of the total $p$-variation of trained weights for any $p\in[1,3]$. Unlike the $C^1$-theory underlying the neural ODE literature, our estimates remain bounded even in the limiting case of weights behaving like Brownian motions, as suggested in [arXiv:2105.12245]. Mathematically, we interpret residual neural network as solutions to (rough) difference equations, and analyse them based on recent results of discrete time signatures and rough path theory.