Differentiable programming and its applications to dynamical systems
This tutorial targets researchers in nonlinear systems, presenting a new programming paradigm but is incremental as it reviews existing concepts without novel results.
The paper introduces differentiable programming as a paradigm combining neural networks with algorithmic modules for enhanced learning capabilities, and reviews its applications and advantages in modeling and predicting dynamical systems.
Differentiable programming is the combination of classical neural networks modules with algorithmic ones in an end-to-end differentiable model. These new models, that use automatic differentiation to calculate gradients, have new learning capabilities (reasoning, attention and memory). In this tutorial, aimed at researchers in nonlinear systems with prior knowledge of deep learning, we present this new programming paradigm, describe some of its new features such as attention mechanisms, and highlight the benefits they bring. Then, we analyse the uses and limitations of traditional deep learning models in the modeling and prediction of dynamical systems. Here, a dynamical system is meant to be a set of state variables that evolve in time under general internal and external interactions. Finally, we review the advantages and applications of differentiable programming to dynamical systems.