Embedding Inequalities for Barron-type Spaces
This work addresses a theoretical gap in machine learning theory for researchers studying approximation and generalization in neural networks, but it is incremental as it builds on existing space definitions.
The paper tackles the problem of clarifying the relationship between Barron and spectral Barron spaces for understanding two-layer neural networks in high dimensions, establishing a continuous embedding with dimension-independent constants and proving tight bounds.
An important problem in machine learning theory is to understand the approximation and generalization properties of two-layer neural networks in high dimensions. To this end, researchers have introduced the Barron space $\mathcal{B}_s(Ω)$ and the spectral Barron space $\mathcal{F}_s(Ω)$, where the index $s\in [0,\infty)$ indicates the smoothness of functions within these spaces and $Ω\subset\mathbb{R}^d$ denotes the input domain. However, the precise relationship between the two types of Barron spaces remains unclear. In this paper, we establish a continuous embedding between them as implied by the following inequality: for any $δ\in (0,1), s\in \mathbb{N}^{+}$ and $f: Ω\mapsto\mathbb{R}$, it holds that \[ δ\|f\|_{\mathcal{F}_{s-δ}(Ω)}\lesssim_s \|f\|_{\mathcal{B}_s(Ω)}\lesssim_s \|f\|_{\mathcal{F}_{s+1}(Ω)}. \] Importantly, the constants do not depend on the input dimension $d$, suggesting that the embedding is effective in high dimensions. Moreover, we also show that the lower and upper bound are both tight.