On minimal representations of shallow ReLU networks
This provides theoretical insights into neural network efficiency for researchers in machine learning theory, but it is incremental as it builds on existing understanding of ReLU networks.
The paper tackles the problem of finding minimal representations for shallow ReLU networks, showing that such networks require either n, n+1, or n+2 neurons depending on the dimensionality, with n+2 needed in higher dimensions, and characterizes the manifold structure of these minimal networks.
The realization function of a shallow ReLU network is a continuous and piecewise affine function $f:\mathbb R^d\to \mathbb R$, where the domain $\mathbb R^{d}$ is partitioned by a set of $n$ hyperplanes into cells on which $f$ is affine. We show that the minimal representation for $f$ uses either $n$, $n+1$ or $n+2$ neurons and we characterize each of the three cases. In the particular case, where the input layer is one-dimensional, minimal representations always use at most $n+1$ neurons but in all higher dimensional settings there are functions for which $n+2$ neurons are needed. Then we show that the set of minimal networks representing $f$ forms a $C^\infty$-submanifold $M$ and we derive the dimension and the number of connected components of $M$. Additionally, we give a criterion for the hyperplanes that guarantees that all continuous, piecewise affine functions are realization functions of appropriate ReLU networks.