Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width One
For deep learning theorists, this work provides tight theoretical bounds on network architecture necessary for universal approximation, resolving a fundamental question in the expressivity of residual networks.
This paper determines the exact minimum block width required for residual neural networks with inner width one to achieve universal approximation, showing that the minimum block width is max(d_x, d_y) for L^p approximation and providing bounds for uniform approximation.
In this paper, we study the universal approximation property of residual neural networks, and obtain some new results. For input and output dimensions $d_x$ and $d_y$, and LeakyReLU, ReLU, ReLU-like activation functions, the upper and lower bounds of the block width are established. To achieve $L^p$ approximation $(1\leq p <+\infty)$ on any compact domain, we show that the exact minimum block width is $\max\{d_x,d_y\}$ when the inner width is 1. Furthermore, we show that residual neural networks with block width $\min\{d_x+d_y, \max\{2d_x+1,d_y\}\}$ can achieve uniform approximation on any compact domain under the constraint that each residual branch has inner width 1. Besides, for any activation function family, we prove that residual neural networks with block width less than $\max\{d_x, d_y\}$ cannot approximate all target functions, both in the $L^p$ sense and the uniform sense, regardless of inner width.