Rates of Approximation by ReLU Shallow Neural Networks
This work addresses a gap in understanding the efficiency of shallow networks for function approximation, which is incremental as it builds on prior deep network results but focuses on the less-studied shallow case.
The paper tackles the problem of approximating functions from Hölder spaces using ReLU shallow neural networks with one hidden layer, showing that they can achieve uniform approximation rates close to optimal, specifically O((log m)^(1/2 + d) * m^(-r/d * (d+2)/(d+4))) for r < d/2 + 2.
Neural networks activated by the rectified linear unit (ReLU) play a central role in the recent development of deep learning. The topic of approximating functions from Hölder spaces by these networks is crucial for understanding the efficiency of the induced learning algorithms. Although the topic has been well investigated in the setting of deep neural networks with many layers of hidden neurons, it is still open for shallow networks having only one hidden layer. In this paper, we provide rates of uniform approximation by these networks. We show that ReLU shallow neural networks with $m$ hidden neurons can uniformly approximate functions from the Hölder space $W_\infty^r([-1, 1]^d)$ with rates $O((\log m)^{\frac{1}{2} +d}m^{-\frac{r}{d}\frac{d+2}{d+4}})$ when $r<d/2 +2$. Such rates are very close to the optimal one $O(m^{-\frac{r}{d}})$ in the sense that $\frac{d+2}{d+4}$ is close to $1$, when the dimension $d$ is large.