Approximation properties of neural ODEs
This work addresses theoretical foundations for neural ODEs, providing insights into their expressiveness and stability, which is incremental as it builds on existing neural ODE research.
The authors tackled the problem of understanding the approximation capabilities of shallow neural networks using neural ODE activation functions, proving universal approximation properties and deriving bounds when constraints like Lipschitz constants and weight norms are applied, showing that constraints reduce expressiveness with quantified accuracy losses.
We study the approximation properties of shallow neural networks whose activation function is defined as the flow map of a neural ordinary differential equation (neural ODE) at the final time of the integration interval. We prove the universal approximation property (UAP) of such shallow neural networks in the space of continuous functions. Furthermore, we investigate the approximation properties of shallow neural networks whose parameters satisfy specific constraints. In particular, we constrain the Lipschitz constant of the neural ODE's flow map and the norms of the weights to increase the network's stability. We prove that the UAP holds if we consider either constraint independently. When both are enforced, there is a loss of expressiveness, and we derive approximation bounds that quantify how accurately such a constrained network can approximate a continuous function.