Deep learning and the rate of approximation by flows
This provides theoretical insights into deep learning approximation mechanisms, but it is incremental as it builds on existing dynamical systems frameworks without empirical validation.
The paper tackles the problem of quantifying the minimal time-horizon needed for deep residual networks to approximate diffeomorphisms via flows, showing it relates to a geodesic distance on a sub-Finsler manifold. This connects learning efficiency to architectural compatibility and suggests a fundamental shift from linear approximation theory to manifold-based dynamics.
We investigate the dependence of the approximation capacity of deep residual networks on its depth in a continuous dynamical systems setting. This can be formulated as the general problem of quantifying the minimal time-horizon required to approximate a diffeomorphism by flows driven by a given family $\mathcal F$ of vector fields. We show that this minimal time can be identified as a geodesic distance on a sub-Finsler manifold of diffeomorphisms, where the local geometry is characterised by a variational principle involving $\mathcal F$. This connects the learning efficiency of target relationships to their compatibility with the learning architectural choice. Further, the results suggest that the key approximation mechanism in deep learning, namely the approximation of functions by composition or dynamics, differs in a fundamental way from linear approximation theory, where linear spaces and norm-based rate estimates are replaced by manifolds and geodesic distances.