Qualitative neural network approximation over R and C: Elementary proofs for analytic and polynomial activation
This work provides theoretical foundations for neural network approximation, which is incremental as it extends existing results using elementary proofs.
The paper tackles the problem of approximating functions using neural networks with analytic or polynomial activation functions, proving that the closure of neural network classes coincides with polynomial spaces for both real and complex cases, and showing that deep networks with polynomial activations can approximate any polynomial under specific width conditions.
In this article, we prove approximation theorems in classes of deep and shallow neural networks with analytic activation functions by elementary arguments. We prove for both real and complex networks with non-polynomial activation that the closure of the class of neural networks coincides with the closure of the space of polynomials. The closure can further be characterized by the Stone-Weierstrass theorem (in the real case) and Mergelyan's theorem (in the complex case). In the real case, we further prove approximation results for networks with higher-dimensional harmonic activation and orthogonally projected linear maps. We further show that fully connected and residual networks of large depth with polynomial activation functions can approximate any polynomial under certain width requirements. All proofs are entirely elementary.