The Representation Power of Neural Networks: Breaking the Curse of Dimensionality
This work is significant for theoretical machine learning researchers interested in the fundamental approximation capabilities of neural networks, showing their efficiency for a specific class of functions.
This paper analyzes the number of neurons and training parameters required for neural networks to approximate Korobov functions, which are multivariate functions with bounded second mixed derivatives. The authors prove upper bounds for shallow and deep neural networks that break the curse of dimensionality, demonstrating near-optimal function approximation.
In this paper, we analyze the number of neurons and training parameters that a neural networks needs to approximate multivariate functions of bounded second mixed derivatives -- Korobov functions. We prove upper bounds on these quantities for shallow and deep neural networks, breaking the curse of dimensionality. Our bounds hold for general activation functions, including ReLU. We further prove that these bounds nearly match the minimal number of parameters any continuous function approximator needs to approximate Korobov functions, showing that neural networks are near-optimal function approximators.