Residual Networks as Geodesic Flows of Diffeomorphisms
This provides theoretical insights into state-of-the-art deep learning architectures for researchers in machine learning, though it is incremental in nature.
The paper tackles the problem of understanding residual networks (ResNets) by showing that their mapping component can be interpreted as the numerical implementation of continuous flows of diffeomorphisms governed by ordinary differential equations, with ResNets with shared weights characterized as approximations of exponential diffeomorphic operators.
This paper addresses the understanding and characterization of residual networks (ResNet), which are among the state-of-the-art deep learning architectures for a variety of supervised learning problems. We focus on the mapping component of ResNets, which map the embedding space towards a new unknown space where the prediction or classification can be stated according to linear criteria. We show that this mapping component can be regarded as the numerical implementation of continuous flows of diffeomorphisms governed by ordinary differential equations. Especially, ResNets with shared weights are fully characterized as numerical approximation of exponential diffeomorphic operators. We stress both theoretically and numerically the relevance of the enforcement of diffeormorphic properties and the importance of numerical issues to make consistent the continuous formulation and the discretized ResNet implementation. We further discuss the resulting theoretical and computational insights on ResNet architectures.