Systematic construction of continuous-time neural networks for linear dynamical systems
This provides a systematic method for researchers and engineers working with linear dynamical systems, though it is incremental as it focuses on a specific subclass.
The paper tackles the challenge of designing neural network architectures for modeling linear time-invariant dynamical systems by proposing a gradient-free algorithm to compute sparse architectures and parameters directly from the system, achieving high accuracy in numerical examples.
Discovering a suitable neural network architecture for modeling complex dynamical systems poses a formidable challenge, often involving extensive trial and error and navigation through a high-dimensional hyper-parameter space. In this paper, we discuss a systematic approach to constructing neural architectures for modeling a subclass of dynamical systems, namely, Linear Time-Invariant (LTI) systems. We use a variant of continuous-time neural networks in which the output of each neuron evolves continuously as a solution of a first-order or second-order Ordinary Differential Equation (ODE). Instead of deriving the network architecture and parameters from data, we propose a gradient-free algorithm to compute sparse architecture and network parameters directly from the given LTI system, leveraging its properties. We bring forth a novel neural architecture paradigm featuring horizontal hidden layers and provide insights into why employing conventional neural architectures with vertical hidden layers may not be favorable. We also provide an upper bound on the numerical errors of our neural networks. Finally, we demonstrate the high accuracy of our constructed networks on three numerical examples.