Designing Universal Causal Deep Learning Models: The Case of Infinite-Dimensional Dynamical Systems from Stochastic Analysis
This work addresses a gap in neural operator design for stochastic analysis, offering a novel framework for approximating causal operators in infinite-dimensional dynamical systems, which is incremental in improving approximation bounds for recurrent networks.
The paper tackles the problem of approximating non-linear operators with temporal structure, such as solution maps to stochastic differential equations, by proposing Causal Neural Operators, a framework that yields universal sequential deep learning models adapted to infinite-dimensional linear metric spaces. The main result shows these models can uniformly approximate Hölder or smooth trace class operators on compact sets across finite-time horizons, with tighter guarantees for recurrent neural networks compared to existing feedforward-based results.
Several non-linear operators in stochastic analysis, such as solution maps to stochastic differential equations, depend on a temporal structure which is not leveraged by contemporary neural operators designed to approximate general maps between Banach space. This paper therefore proposes an operator learning solution to this open problem by introducing a deep learning model-design framework that takes suitable infinite-dimensional linear metric spaces, e.g. Banach spaces, as inputs and returns a universal \textit{sequential} deep learning model adapted to these linear geometries specialized for the approximation of operators encoding a temporal structure. We call these models \textit{Causal Neural Operators}. Our main result states that the models produced by our framework can uniformly approximate on compact sets and across arbitrarily finite-time horizons Hölder or smooth trace class operators, which causally map sequences between given linear metric spaces. Our analysis uncovers new quantitative relationships on the latent state-space dimension of Causal Neural Operators, which even have new implications for (classical) finite-dimensional Recurrent Neural Networks. In addition, our guarantees for recurrent neural networks are tighter than the available results inherited from feedforward neural networks when approximating dynamical systems between finite-dimensional spaces.