Nonlocal techniques for the analysis of deep ReLU neural network approximations
This provides theoretical foundations for neural network approximations in high-dimensional function spaces, though it is incremental as it builds on prior work.
The paper tackles the problem of approximating functions from Sobolev and Barron classes using deep ReLU neural networks, showing that a specific piece-wise linear system serves as a Riesz basis for these classes and enabling proofs that avoid the curse of dimension with tracked implicit constants.
Recently, Daubechies, DeVore, Foucart, Hanin, and Petrova introduced a system of piece-wise linear functions, which can be easily reproduced by artificial neural networks with the ReLU activation function and which form a Riesz basis of $L_2([0,1])$. This work was generalized by two of the authors to the multivariate setting. We show that this system serves as a Riesz basis also for Sobolev spaces $W^s([0,1]^d)$ and Barron classes ${\mathbb B}^s([0,1]^d)$ with smoothness $0<s<1$. We apply this fact to re-prove some recent results on the approximation of functions from these classes by deep neural networks. Our proof method avoids using local approximations and allows us to track also the implicit constants as well as to show that we can avoid the curse of dimension. Moreover, we also study how well one can approximate Sobolev and Barron functions by ANNs if only function values are known.