Weighted variation spaces and approximation by shallow ReLU networks
This work addresses a foundational issue in neural network approximation theory by refining model class definitions for domain-specific applications, though it is incremental as it builds on existing classes like Barron and variation spaces.
The paper tackles the problem of approximating functions on bounded domains using shallow ReLU neural networks by introducing weighted variation spaces as new model classes that are domain-specific. It shows that these classes are strictly larger than classical domain-independent ones while maintaining the same approximation rates.
We investigate the approximation of functions $f$ on a bounded domain $Ω\subset \mathbb{R}^d$ by the outputs of single-hidden-layer ReLU neural networks of width $n$. This form of nonlinear $n$-term dictionary approximation has been intensely studied since it is the simplest case of neural network approximation (NNA). There are several celebrated approximation results for this form of NNA that introduce novel model classes of functions on $Ω$ whose approximation rates do not grow unbounded with the input dimension. These novel classes include Barron classes, and classes based on sparsity or variation such as the Radon-domain BV classes. The present paper is concerned with the definition of these novel model classes on domains $Ω$. The current definition of these model classes does not depend on the domain $Ω$. A new and more proper definition of model classes on domains is given by introducing the concept of weighted variation spaces. These new model classes are intrinsic to the domain itself. The importance of these new model classes is that they are strictly larger than the classical (domain-independent) classes. Yet, it is shown that they maintain the same NNA rates.