How do infinite width bounded norm networks look in function space?
This provides foundational insights into the function space of neural networks, addressing a theoretical bottleneck for researchers in machine learning theory.
The paper tackles the problem of characterizing the functions representable by infinite-width ReLU networks with bounded Euclidean norm, showing that for single-hidden-layer networks approximating univariate functions, the minimal required norm is max(∫|f''(x)|dx, |f'(-∞) + f'(+∞)|), leading to linear spline interpolation as the minimal norm fit for sampled data.
We consider the question of what functions can be captured by ReLU networks with an unbounded number of units (infinite width), but where the overall network Euclidean norm (sum of squares of all weights in the system, except for an unregularized bias term for each unit) is bounded; or equivalently what is the minimal norm required to approximate a given function. For functions $f : \mathbb R \rightarrow \mathbb R$ and a single hidden layer, we show that the minimal network norm for representing $f$ is $\max(\int |f''(x)| dx, |f'(-\infty) + f'(+\infty)|)$, and hence the minimal norm fit for a sample is given by a linear spline interpolation.