Condition Numbers and Eigenvalue Spectra of Shallow Networks on Spheres
This work addresses numerical stability issues in neural network approximation on spheres, which is incremental as it builds on existing spectral analysis methods.
The paper tackles the problem of estimating condition numbers and eigenvalue spectra for shallow ReLU^k neural networks on spheres, showing that under antipodally quasi-uniform conditions, the condition number is sharp and linking the smallest eigenvalues to low-degree polynomials and the largest to high-degree polynomials.
We present an estimation of the condition numbers of the \emph{mass} and \emph{stiffness} matrices arising from shallow ReLU$^k$ neural networks defined on the unit sphere~$\mathbb{S}^d$. In particular, when $\{θ_j^*\}_{j=1}^n \subset \mathbb{S}^d$ is \emph{antipodally quasi-uniform}, the condition number is sharp. Indeed, in this case, we obtain sharp asymptotic estimates for the full spectrum of eigenvalues and characterize the structure of the corresponding eigenspaces, showing that the smallest eigenvalues are associated with an eigenbasis of low-degree polynomials while the largest eigenvalues are linked to high-degree polynomials. This spectral analysis establishes a precise correspondence between the approximation power of the network and its numerical stability.