Approximation Rates of Shallow Neural Networks: Barron Spaces, Activation Functions and Optimality Analysis
This work addresses the theoretical limits of shallow neural networks for function approximation, which is relevant for researchers in machine learning theory and neural network design.
The paper investigates the approximation properties of shallow neural networks with ReLU^k activation functions in Barron and Sobolev spaces, proving that optimal rates are unattainable under certain conditions and establishing rates that confirm the curse of dimensionality.
This paper investigates the approximation properties of shallow neural networks with activation functions that are powers of exponential functions. It focuses on the dependence of the approximation rate on the dimension and the smoothness of the function being approximated within the Barron function space. We examine the approximation rates of ReLU$^{k}$ activation functions, proving that the optimal rate cannot be achieved under $\ell^{1}$-bounded coefficients or insufficient smoothness conditions. We also establish optimal approximation rates in various norms for functions in Barron spaces and Sobolev spaces, confirming the curse of dimensionality. Our results clarify the limits of shallow neural networks' approximation capabilities and offer insights into the selection of activation functions and network structures.