Deep Network Approximation for Smooth Functions
It provides theoretical guarantees for deep learning approximation, which is foundational for understanding neural network capabilities in machine learning.
This paper tackles the problem of approximating smooth functions using deep ReLU networks by establishing nearly optimal error bounds in terms of width and depth, achieving an approximation error of O(||f||_C^s N^{-2s/d} L^{-2s/d}) for functions in C^s([0,1]^d).
This paper establishes the (nearly) optimal approximation error characterization of deep rectified linear unit (ReLU) networks for smooth functions in terms of both width and depth simultaneously. To that end, we first prove that multivariate polynomials can be approximated by deep ReLU networks of width $\mathcal{O}(N)$ and depth $\mathcal{O}(L)$ with an approximation error $\mathcal{O}(N^{-L})$. Through local Taylor expansions and their deep ReLU network approximations, we show that deep ReLU networks of width $\mathcal{O}(N\ln N)$ and depth $\mathcal{O}(L\ln L)$ can approximate $f\in C^s([0,1]^d)$ with a nearly optimal approximation error $\mathcal{O}(\|f\|_{C^s([0,1]^d)}N^{-2s/d}L^{-2s/d})$. Our estimate is non-asymptotic in the sense that it is valid for arbitrary width and depth specified by $N\in\mathbb{N}^+$ and $L\in\mathbb{N}^+$, respectively.