A note on regularised NTK dynamics with an application to PAC-Bayesian training
This work provides a theoretical framework for analyzing wide neural networks trained with regularization, potentially aiding in understanding generalization, but it is incremental as it builds on existing NTK theory.
The paper tackles the problem of understanding the training dynamics of neural networks under regularization that keeps parameters close to their initialization, showing that in the infinite-width limit, the dynamics are governed by the neural tangent kernel with an additional term due to regularization, which can be applied to study PAC-Bayes bounds for generalization.
We establish explicit dynamics for neural networks whose training objective has a regularising term that constrains the parameters to remain close to their initial value. This keeps the network in a lazy training regime, where the dynamics can be linearised around the initialisation. The standard neural tangent kernel (NTK) governs the evolution during the training in the infinite-width limit, although the regularisation yields an additional term appears in the differential equation describing the dynamics. This setting provides an appropriate framework to study the evolution of wide networks trained to optimise generalisation objectives such as PAC-Bayes bounds, and hence potentially contribute to a deeper theoretical understanding of such networks.