Deep Neural Networks Inspired by Differential Equations
This is an incremental review that synthesizes existing research on using differential equations to improve neural networks, potentially aiding researchers in developing more interpretable and generalizable AI methods.
The paper reviews deep neural network architectures and dynamic modeling methods inspired by differential equations, such as ODEs and SDEs, to address challenges in theoretical understanding, interpretability, and generalization, presenting numerical comparisons to illustrate their characteristics and performance.
Deep learning has become a pivotal technology in fields such as computer vision, scientific computing, and dynamical systems, significantly advancing these disciplines. However, neural Networks persistently face challenges related to theoretical understanding, interpretability, and generalization. To address these issues, researchers are increasingly adopting a differential equations perspective to propose a unified theoretical framework and systematic design methodologies for neural networks. In this paper, we provide an extensive review of deep neural network architectures and dynamic modeling methods inspired by differential equations. We specifically examine deep neural network models and deterministic dynamical network constructs based on ordinary differential equations (ODEs), as well as regularization techniques and stochastic dynamical network models informed by stochastic differential equations (SDEs). We present numerical comparisons of these models to illustrate their characteristics and performance. Finally, we explore promising research directions in integrating differential equations with deep learning to offer new insights for developing intelligent computational methods that boast enhanced interpretability and generalization capabilities.