Residual Networks: Lyapunov Stability and Convex Decomposition
This addresses the training stability and generalization issues in deep learning, particularly for residual networks, though it appears incremental in its architectural extension.
The paper tackles the problem of training deep neural networks where error typically degrades with increasing depth, showing that residual networks maintain stability due to Lyapunov stability in gradient descent. It presents an architecture using paired residual networks to decompose functions into convex and concave parts, achieving solutions with small Lipschitz constants that prevent overfitting.
While training error of most deep neural networks degrades as the depth of the network increases, residual networks appear to be an exception. We show that the main reason for this is the Lyapunov stability of the gradient descent algorithm: for an arbitrarily chosen step size, the equilibria of the gradient descent are most likely to remain stable for the parametrization of residual networks. We then present an architecture with a pair of residual networks to approximate a large class of functions by decomposing them into a convex and a concave part. Some parameters of this model are shown to change little during training, and this imperfect optimization prevents overfitting the data and leads to solutions with small Lipschitz constants, while providing clues about the generalization of other deep networks.