Finite-time Lyapunov exponents of deep neural networks
This work provides insights into the fundamental mechanisms of deep learning for researchers in machine learning and dynamical systems, though it is incremental in applying existing dynamical systems concepts to neural networks.
The authors tackled the problem of understanding how deep neural networks respond to small input perturbations by computing finite-time Lyapunov exponents, revealing that maximal exponents form geometric structures in input space that separate classes and visualize the network's learned geometry.
We compute how small input perturbations affect the output of deep neural networks, exploring an analogy between deep networks and dynamical systems, where the growth or decay of local perturbations is characterised by finite-time Lyapunov exponents. We show that the maximal exponent forms geometrical structures in input space, akin to coherent structures in dynamical systems. Ridges of large positive exponents divide input space into different regions that the network associates with different classes. These ridges visualise the geometry that deep networks construct in input space, shedding light on the fundamental mechanisms underlying their learning capabilities.