Deep, Skinny Neural Networks are not Universal Approximators
This provides foundational insights for researchers and practitioners in machine learning, addressing a core theoretical limitation in neural network design.
The paper tackles the problem of understanding architectural limitations in neural networks by examining topological constraints on level sets, showing that deep, skinny networks are not universal approximators regardless of depth for a broad family of activation functions.
In order to choose a neural network architecture that will be effective for a particular modeling problem, one must understand the limitations imposed by each of the potential options. These limitations are typically described in terms of information theoretic bounds, or by comparing the relative complexity needed to approximate example functions between different architectures. In this paper, we examine the topological constraints that the architecture of a neural network imposes on the level sets of all the functions that it is able to approximate. This approach is novel for both the nature of the limitations and the fact that they are independent of network depth for a broad family of activation functions.