Universal Approximation Properties for an ODENet and a ResNet: Mathematical Analysis and Numerical Experiments
This work provides a theoretical foundation for the approximation capabilities of ODENets and ResNets, which is significant for researchers and practitioners in deep learning seeking to understand the expressive power of these architectures.
This paper proves that ODENets of width n+m and ResNets (as depth approaches infinity) can approximate any continuous function on a compact subset of R^n, given a non-polynomial continuous activation function. They also derive a gradient for a loss function and apply it to regression and classification tasks on MNIST.
We prove a universal approximation property (UAP) for a class of ODENet and a class of ResNet, which are simplified mathematical models for deep learning systems with skip connections. The UAP can be stated as follows. Let $n$ and $m$ be the dimension of input and output data, and assume $m\leq n$. Then we show that ODENet of width $n+m$ with any non-polynomial continuous activation function can approximate any continuous function on a compact subset on $\mathbb{R}^n$. We also show that ResNet has the same property as the depth tends to infinity. Furthermore, we derive the gradient of a loss function explicitly with respect to a certain tuning variable. We use this to construct a learning algorithm for ODENet. To demonstrate the usefulness of this algorithm, we apply it to a regression problem, a binary classification, and a multinomial classification in MNIST.