Mathematical Foundations of Neural Tangents and Infinite-Width Networks
This work provides a comprehensive mathematical framework for connecting infinite-width theory to practical deep learning, addressing a foundational gap in neural network analysis.
The authors tackled the problem of bridging infinite-width neural network theory with practical architectures by proposing the NTK-ECRN, which integrates Fourier features, residual connections, and stochastic depth to enable rigorous analysis of kernel evolution during training. Their theoretical contributions include deriving bounds on NTK dynamics and linking spectral properties to generalization, with empirical results validating improved training stability and generalization on synthetic and benchmark datasets.
We investigate the mathematical foundations of neural networks in the infinite-width regime through the Neural Tangent Kernel (NTK). We propose the NTK-Eigenvalue-Controlled Residual Network (NTK-ECRN), an architecture integrating Fourier feature embeddings, residual connections with layerwise scaling, and stochastic depth to enable rigorous analysis of kernel evolution during training. Our theoretical contributions include deriving bounds on NTK dynamics, characterizing eigenvalue evolution, and linking spectral properties to generalization and optimization stability. Empirical results on synthetic and benchmark datasets validate the predicted kernel behavior and demonstrate improved training stability and generalization. This work provides a comprehensive framework bridging infinite-width theory and practical deep-learning architectures.