Sharp Lower Bounds for Linearized ReLU^k Approximation on the Sphere
This work provides foundational theoretical limits for neural network approximation, showing that the advantage of ReLU^k networks over finite elements is intrinsically limited, which is significant for researchers in approximation theory and machine learning.
The paper tackles the problem of approximating functions on the unit sphere using linearized shallow ReLU^k neural networks, proving a sharp lower bound of order n^{-(d+2k+1)/(2d)} for convergence when the target function has sufficient smoothness, which matches existing upper bounds and establishes the exact saturation order.
We prove a saturation theorem for linearized shallow ReLU$^k$ neural networks on the unit sphere $\mathbb S^d$. For any antipodally quasi-uniform set of centers, if the target function has smoothness $r>\tfrac{d+2k+1}{2}$, then the best $\mathcal{L}^2(\mathbb S^d)$ approximation cannot converge faster than order $n^{-\frac{d+2k+1}{2d}}$. This lower bound matches existing upper bounds, thereby establishing the exact saturation order $\tfrac{d+2k+1}{2d}$ for such networks. Our results place linearized neural-network approximation firmly within the classical saturation framework and show that, although ReLU$^k$ networks outperform finite elements under equal degrees $k$, this advantage is intrinsically limited.