Connections between physics, mathematics and deep learning
This work provides a formal bridge between deep learning and physics/mathematics, potentially encouraging interdisciplinary collaboration, but it is incremental in applying existing theoretical frameworks to neural networks.
The paper tackles the problem of deriving neural network equations in a coordinate-invariant way by connecting deep learning to physics principles like Fermat's principle and differential geometry, showing that the loss function acts as a Hamiltonian and introducing a layer metric for pretraining.
Starting from the Fermat's principle of least action, which governs classical and quantum mechanics and from the theory of exterior differential forms, which governs the geometry of curved manifolds, we show how to derive the equations governing neural networks in an intrinsic, coordinate invariant way, where the loss function plays the role of the Hamiltonian. To be covariant, these equations imply a layer metric which is instrumental in pretraining and explains the role of conjugation when using complex numbers. The differential formalism also clarifies the relation of the gradient descent optimizer with Aristotelian and Newtonian mechanics and why large learning steps break the logic of the linearization procedure. We hope that this formal presentation of the differential geometry of neural networks will encourage some physicists to dive into deep learning, and reciprocally, that the specialists of deep learning will better appreciate the close interconnection of their subject with the foundations of classical and quantum field theory.