Understanding the Theoretical Foundations of Deep Neural Networks through Differential Equations
This addresses the problem of systematic development in deep learning for researchers and practitioners, but it is incremental as it surveys existing ideas rather than introducing new methods.
This survey tackles the lack of a principled theoretical foundation for deep neural networks by presenting differential equations as a framework for understanding, analyzing, and improving them, organizing the discussion around guiding questions and perspectives at model and layer levels.
Deep neural networks (DNNs) have achieved remarkable empirical success, yet the absence of a principled theoretical foundation continues to hinder their systematic development. In this survey, we present differential equations as a theoretical foundation for understanding, analyzing, and improving DNNs. We organize the discussion around three guiding questions: i) how differential equations offer a principled understanding of DNN architectures, ii) how tools from differential equations can be used to improve DNN performance in a principled way, and iii) what real-world applications benefit from grounding DNNs in differential equations. We adopt a two-fold perspective spanning the model level, which interprets the whole DNN as a differential equation, and the layer level, which models individual DNN components as differential equations. From these two perspectives, we review how this framework connects model design, theoretical analysis, and performance improvement. We further discuss real-world applications, as well as key challenges and opportunities for future research.