Label-NTK Alignments and A Tighter Convergence Bound in the NTK Regime
Provides tighter convergence guarantees for over-parameterized neural networks, addressing the gap between theory and practice in optimization speed.
The paper identifies Label-NTK and Residual-NTK alignment phenomena, showing that label and residual projections onto NTK eigenvectors scale with eigenvalues, leading to a refined convergence bound that closely matches practical training dynamics and significantly improves over classical worst-case results.
The Neural Tangent Kernel (NTK) framework explains optimization in over-parameterized neural networks via approximately linearized dynamics, yielding exponential convergence guarantees. However, existing results are often overly pessimistic and do not match the fast training in practice, as they depend on the smallest NTK eigenvalue, which is typically extremely small in practice. In this work, we develop sharper convergence guarantees by characterizing the interaction between data labels and the NTK eigen-spectrum. We identify two key phenomena, Label-NTK alignment and Residual-NTK alignment, showing that projections of labels and residuals onto NTK eigenvectors scale with the corresponding eigenvalues. We provide empirical evidence and theoretical justification under mild data assumptions. Exploiting these alignment properties, we derive a refined convergence bound that depends on the full spectrum and closely matches practical training dynamics, significantly improving over classical worst-case results. We further obtain improved generalization bounds. Experiments on MLPs and CNNs across multiple datasets validate our theory.