Approximation and learning of anisotropic and mixed smooth functions by deep ReLU neural networks
This work is significant for researchers in approximation theory and deep learning, as it provides theoretical guarantees on the efficiency of deep ReLU networks for a broader range of smooth functions, potentially informing network design for high-dimensional data.
This paper investigates the approximation capabilities of deep ReLU neural networks for anisotropic and mixed smooth functions. It establishes an approximation rate of O((WL)^(-2ñs)) for anisotropic Besov spaces and O((WL)^(-2s)) up to logarithmic factors for mixed smooth Besov spaces, where ñs is the mean smoothness. These results demonstrate that deep ReLU networks can achieve minimax optimal rates for various smooth function classes.
This paper studies how efficiently deep ReLU neural networks can approximate and learn smooth functions. When the error is measured in $L^p([0,1]^d)$ norm and the approximator is a network with width $W$ and depth $L$, recent works have proven the supper approximation rate $\mathcal{O}((WL)^{-2s/d})$ for Besov space $\mathcal{B}^s_{q,r}([0,1]^d)$ under the Sobolev embedding condition $s/d>1/q-1/p$. In order to overcome the curse of dimensionality in this rate, we extent this result to anisotropic and mixed smooth function classes. We establish the approximation rate $\mathcal{O}((WL)^{-2\tilde{s}})$ for anisotropic Besov space $\mathcal{B}^{\boldsymbol{s}}_{q,r}([0,1]^d)$ with anisotropic smoothness $\boldsymbol{s}=(s_1,\dots,s_d)$ under the embedding condition $\tilde{s} > 1/q-1/p$, where the mean smoothness $\tilde{s} = (\sum_{i=1}^d s_i^{-1})^{-1}$. For mixed smooth Besov space $\mathcal{MB}^s_{q,r}([0,1]^d)$ with mixed smoothness $s>1/q-1/p$, we show that the approximation rate $\mathcal{O}((WL)^{-2s})$ holds up to logarithmic factors. Using these results, we also derive approximation bounds for the composition of anisotropic Besov functions. As an application, it is shown that deep ReLU neural networks can achieve minimax optimal rates up to logarithmic factors for a wide range of smooth function classes.