Critical Learning Periods in Deep Neural Networks
This work addresses the under-scrutinized role of initial learning transients in training outcomes for machine learning researchers, though it is incremental as it builds on existing biological and theoretical insights.
The study investigates critical learning periods in deep neural networks, showing that temporary stimulus deficits during early training can impair skill development, with impairment severity depending on deficit timing, duration, and network size, and deficits not affecting low-level statistics can be overcome with further training. It finds that Fisher Information rises early then decreases, indicating a loss of 'Information Plasticity,' and suggests the first few epochs are critical for forming optimal connections that remain unchanged later.
Similar to humans and animals, deep artificial neural networks exhibit critical periods during which a temporary stimulus deficit can impair the development of a skill. The extent of the impairment depends on the onset and length of the deficit window, as in animal models, and on the size of the neural network. Deficits that do not affect low-level statistics, such as vertical flipping of the images, have no lasting effect on performance and can be overcome with further training. To better understand this phenomenon, we use the Fisher Information of the weights to measure the effective connectivity between layers of a network during training. Counterintuitively, information rises rapidly in the early phases of training, and then decreases, preventing redistribution of information resources in a phenomenon we refer to as a loss of "Information Plasticity". Our analysis suggests that the first few epochs are critical for the creation of strong connections that are optimal relative to the input data distribution. Once such strong connections are created, they do not appear to change during additional training. These findings suggest that the initial learning transient, under-scrutinized compared to asymptotic behavior, plays a key role in determining the outcome of the training process. Our findings, combined with recent theoretical results in the literature, also suggest that forgetting (decrease of information in the weights) is critical to achieving invariance and disentanglement in representation learning. Finally, critical periods are not restricted to biological systems, but can emerge naturally in learning systems, whether biological or artificial, due to fundamental constrains arising from learning dynamics and information processing.