ResNet with one-neuron hidden layers is a Universal Approximator
This provides a theoretical foundation for the representational power of ResNets in deep learning, addressing a fundamental limitation in neural network architecture for researchers and practitioners.
The paper tackles the problem of whether narrow deep networks can universally approximate functions, showing that a very deep ResNet with one-neuron hidden layers and ReLU activations can uniformly approximate any Lebesgue integrable function in d dimensions, in contrast to fully connected networks which fail at width d.
We demonstrate that a very deep ResNet with stacked modules with one neuron per hidden layer and ReLU activation functions can uniformly approximate any Lebesgue integrable function in $d$ dimensions, i.e. $\ell_1(\mathbb{R}^d)$. Because of the identity mapping inherent to ResNets, our network has alternating layers of dimension one and $d$. This stands in sharp contrast to fully connected networks, which are not universal approximators if their width is the input dimension $d$ [Lu et al, 2017; Hanin and Sellke, 2017]. Hence, our result implies an increase in representational power for narrow deep networks by the ResNet architecture.