Advantage of Deep Neural Networks for Estimating Functions with Singularity on Hypersurfaces
This provides a theoretical explanation for the advantage of deep neural networks in handling complex, non-smooth functions, which is incremental but clarifies a gap in existing minimax rate analyses.
The paper tackles the problem of estimating non-smooth functions with singularities on hypersurfaces in nonparametric regression, showing that deep neural networks achieve an almost optimal convergence rate and outperform standard methods like kernel and harmonic analysis-based estimators in specific scenarios.
We develop a minimax rate analysis to describe the reason that deep neural networks (DNNs) perform better than other standard methods. For nonparametric regression problems, it is well known that many standard methods attain the minimax optimal rate of estimation errors for smooth functions, and thus, it is not straightforward to identify the theoretical advantages of DNNs. This study tries to fill this gap by considering the estimation for a class of non-smooth functions that have singularities on hypersurfaces. Our findings are as follows: (i) We derive the generalization error of a DNN estimator and prove that its convergence rate is almost optimal. (ii) We elucidate a phase diagram of estimation problems, which describes the situations where the DNNs outperform a general class of estimators, including kernel methods, Gaussian process methods, and others. We additionally show that DNNs outperform harmonic analysis based estimators. This advantage of DNNs comes from the fact that a shape of singularity can be successfully handled by their multi-layered structure.