On Sharpness of Error Bounds for Multivariate Neural Network Approximation
This work addresses the theoretical understanding of approximation errors in neural networks for researchers in mathematical analysis and machine learning, but it is incremental as it extends prior univariate results to the multivariate case.
The paper tackles the problem of establishing error bounds for multivariate neural network approximation using sums of ridge functions, and it proves that these bounds are best possible by constructing counterexamples for specific activation functions like the logistic and piecewise polynomial ones.
Single hidden layer feedforward neural networks can represent multivariate functions that are sums of ridge functions. These ridge functions are defined via an activation function and customizable weights. The paper deals with best non-linear approximation by such sums of ridge functions. Error bounds are presented in terms of moduli of smoothness. The main focus, however, is to prove that the bounds are best possible. To this end, counterexamples are constructed with a non-linear, quantitative extension of the uniform boundedness principle. They show sharpness with respect to Lipschitz classes for the logistic activation function and for certain piecewise polynomial activation functions. The paper is based on univariate results in (Goebbels, St.: On sharpness of error bounds for univariate approximation by single hidden layer feedforward neural networks. Results Math 75 (3), 2020, article 109, https://rdcu.be/b5mKH).