Optimal Approximation Complexity of High-Dimensional Functions with Neural Networks
This work addresses the fundamental challenge of function approximation in machine learning, providing theoretical guarantees for neural network efficiency, though it is incremental by building on prior results.
The paper tackles the problem of approximating high-dimensional functions with neural networks by showing that networks using ReLU and x^2 activations achieve optimal approximation rates for analytic and Sobolev functions, overcoming the curse of dimensionality in some contexts.
We investigate properties of neural networks that use both ReLU and $x^2$ as activation functions and build upon previous results to show that both analytic functions and functions in Sobolev spaces can be approximated by such networks of constant depth to arbitrary accuracy, demonstrating optimal order approximation rates across all nonlinear approximators, including standard ReLU networks. We then show how to leverage low local dimensionality in some contexts to overcome the curse of dimensionality, obtaining approximation rates that are optimal for unknown lower-dimensional subspaces.