Arbitrary-Depth Universal Approximation Theorems for Operator Neural Networks
This addresses a theoretical gap in operator neural network approximation theory, showing depth can compensate for width limitations, which is incremental but foundational for understanding network architectures.
The paper proves that operator neural networks with bounded width and arbitrary depth can universally approximate continuous nonlinear operators, specifically achieving this with width five for certain activation functions. It also demonstrates a depth separation result where deep networks require exponentially wide shallow networks to approximate them.
The standard Universal Approximation Theorem for operator neural networks (NNs) holds for arbitrary width and bounded depth. Here, we prove that operator NNs of bounded width and arbitrary depth are universal approximators for continuous nonlinear operators. In our main result, we prove that for non-polynomial activation functions that are continuously differentiable at a point with a nonzero derivative, one can construct an operator NN of width five, whose inputs are real numbers with finite decimal representations, that is arbitrarily close to any given continuous nonlinear operator. We derive an analogous result for non-affine polynomial activation functions. We also show that depth has theoretical advantages by constructing operator ReLU NNs of depth $2k^3+8$ and constant width that cannot be well-approximated by any operator ReLU NN of depth $k$, unless its width is exponential in $k$.