A deep network construction that adapts to intrinsic dimensionality beyond the domain
This work addresses the challenge of function approximation in high-dimensional spaces for machine learning, showing that deep networks can adapt to intrinsic dimensionality, potentially relaxing the low-dimensional manifold assumption.
The paper tackles the problem of approximating two-layer compositions f(x)=g(φ(x)) via deep ReLU networks, achieving near-optimal rates that depend only on the complexity of the dimensionality-reducing map φ, not the ambient dimension.
We study the approximation of two-layer compositions $f(x) = g(φ(x))$ via deep networks with ReLU activation, where $φ$ is a geometrically intuitive, dimensionality reducing feature map. We focus on two intuitive and practically relevant choices for $φ$: the projection onto a low-dimensional embedded submanifold and a distance to a collection of low-dimensional sets. We achieve near optimal approximation rates, which depend only on the complexity of the dimensionality reducing map $φ$ rather than the ambient dimension. Since $φ$ encapsulates all nonlinear features that are material to the function $f$, this suggests that deep nets are faithful to an intrinsic dimension governed by $f$ rather than the complexity of the domain of $f$. In particular, the prevalent assumption of approximating functions on low-dimensional manifolds can be significantly relaxed using functions of type $f(x) = g(φ(x))$ with $φ$ representing an orthogonal projection onto the same manifold.