Universal Approximation Property of Hamiltonian Deep Neural Networks
This work addresses the lack of theoretical understanding for HDNNs, which are used in applications requiring stable training, but it is incremental as it builds on existing HDNN frameworks.
The paper tackles the problem of quantifying the expressivity of Hamiltonian Deep Neural Networks (HDNNs) by proving a universal approximation theorem, showing that HDNNs can approximate any continuous function over a compact domain, which provides a theoretical foundation for their practical use.
This paper investigates the universal approximation capabilities of Hamiltonian Deep Neural Networks (HDNNs) that arise from the discretization of Hamiltonian Neural Ordinary Differential Equations. Recently, it has been shown that HDNNs enjoy, by design, non-vanishing gradients, which provide numerical stability during training. However, although HDNNs have demonstrated state-of-the-art performance in several applications, a comprehensive study to quantify their expressivity is missing. In this regard, we provide a universal approximation theorem for HDNNs and prove that a portion of the flow of HDNNs can approximate arbitrary well any continuous function over a compact domain. This result provides a solid theoretical foundation for the practical use of HDNNs.