The Quenching-Activation Behavior of the Gradient Descent Dynamics for Two-layer Neural Network Models
This addresses the problem of understanding training dynamics in neural networks for researchers, offering insights into implicit regularization, though it appears incremental as it builds on existing studies of gradient descent behavior.
The study investigated gradient descent training dynamics for two-layer neural networks, identifying a quenching-activation process where neurons split into activated and quenched groups, providing a mechanism for implicit regularization.
A numerical and phenomenological study of the gradient descent (GD) algorithm for training two-layer neural network models is carried out for different parameter regimes when the target function can be accurately approximated by a relatively small number of neurons. It is found that for Xavier-like initialization, there are two distinctive phases in the dynamic behavior of GD in the under-parametrized regime: An early phase in which the GD dynamics follows closely that of the corresponding random feature model and the neurons are effectively quenched, followed by a late phase in which the neurons are divided into two groups: a group of a few "activated" neurons that dominate the dynamics and a group of background (or "quenched") neurons that support the continued activation and deactivation process. This neural network-like behavior is continued into the mildly over-parametrized regime, where it undergoes a transition to a random feature-like behavior. The quenching-activation process seems to provide a clear mechanism for "implicit regularization". This is qualitatively different from the dynamics associated with the "mean-field" scaling where all neurons participate equally and there does not appear to be qualitative changes when the network parameters are changed.