A unified framework for Hamiltonian deep neural networks
This work addresses training difficulties in deep neural networks for machine learning practitioners, but it is incremental as it builds on existing Hamiltonian-inspired methods.
The authors tackled the problem of vanishing/exploding gradients in deep neural networks by proposing a Hamiltonian framework derived from time discretization, which ensures marginal stability to mitigate gradient issues, and demonstrated its performance on benchmark classification tasks like MNIST digit recognition.
Training deep neural networks (DNNs) can be difficult due to the occurrence of vanishing/exploding gradients during weight optimization. To avoid this problem, we propose a class of DNNs stemming from the time discretization of Hamiltonian systems. The time-invariant version of the corresponding Hamiltonian models enjoys marginal stability, a property that, as shown in previous works and for specific DNNs architectures, can mitigate convergence to zero or divergence of gradients. In the present paper, we formally study this feature by deriving and analysing the backward gradient dynamics in continuous time. The proposed Hamiltonian framework, besides encompassing existing networks inspired by marginally stable ODEs, allows one to derive new and more expressive architectures. The good performance of the novel DNNs is demonstrated on benchmark classification problems, including digit recognition using the MNIST dataset.