Optimal rates of approximation by shallow ReLU$^k$ neural networks and applications to nonparametric regression
This provides theoretical guarantees for neural network efficiency in nonparametric regression, complementing existing deep network results, but is incremental as it extends known approximation theory to specific architectures.
The paper establishes optimal approximation rates for shallow ReLU^k neural networks in terms of neuron count and applies these to derive convergence rates for nonparametric regression, showing that shallow networks achieve minimax optimal rates for learning Hölder functions and over-parameterized networks achieve nearly optimal rates.
We study the approximation capacity of some variation spaces corresponding to shallow ReLU$^k$ neural networks. It is shown that sufficiently smooth functions are contained in these spaces with finite variation norms. For functions with less smoothness, the approximation rates in terms of the variation norm are established. Using these results, we are able to prove the optimal approximation rates in terms of the number of neurons for shallow ReLU$^k$ neural networks. It is also shown how these results can be used to derive approximation bounds for deep neural networks and convolutional neural networks (CNNs). As applications, we study convergence rates for nonparametric regression using three ReLU neural network models: shallow neural network, over-parameterized neural network, and CNN. In particular, we show that shallow neural networks can achieve the minimax optimal rates for learning Hölder functions, which complements recent results for deep neural networks. It is also proven that over-parameterized (deep or shallow) neural networks can achieve nearly optimal rates for nonparametric regression.