Deep ReLU network approximation of functions on a manifold
This work addresses the challenge of function approximation on manifolds for machine learning applications, providing theoretical guarantees for deep networks in this setting.
The paper tackles the problem of approximating Hölder functions on a manifold using deep ReLU networks, showing that sparsely connected networks achieve an error of ε with roughly ε^{-d*/β} log(1/ε) non-zero parameters. As an application, it derives statistical convergence rates for an empirical risk minimization estimator.
Whereas recovery of the manifold from data is a well-studied topic, approximation rates for functions defined on manifolds are less known. In this work, we study a regression problem with inputs on a $d^*$-dimensional manifold that is embedded into a space with potentially much larger ambient dimension. It is shown that sparsely connected deep ReLU networks can approximate a Hölder function with smoothness index $β$ up to error $ε$ using of the order of $ε^{-d^*/β}\log(1/ε)$ many non-zero network parameters. As an application, we derive statistical convergence rates for the estimator minimizing the empirical risk over all possible choices of bounded network parameters.