Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimension
This provides more general theoretical guarantees for neural network optimization and memorization, applicable to a wider range of data scenarios, though it is incremental in extending existing bounds.
The paper tackles the problem of bounding the smallest eigenvalue of the neural tangent kernel (NTK) for arbitrary spherical data without distributional assumptions or high-dimensional scaling, achieving bounds that hold with high probability even when input dimension is constant relative to sample size, based on a measure of data collinearity.
Bounds on the smallest eigenvalue of the neural tangent kernel (NTK) are a key ingredient in the analysis of neural network optimization and memorization. However, existing results require distributional assumptions on the data and are limited to a high-dimensional setting, where the input dimension $d_0$ scales at least logarithmically in the number of samples $n$. In this work we remove both of these requirements and instead provide bounds in terms of a measure of the collinearity of the data: notably these bounds hold with high probability even when $d_0$ is held constant versus $n$. We prove our results through a novel application of the hemisphere transform.