Investigating Learning in Deep Neural Networks using Layer-Wise Weight Change
This work addresses the problem of understanding neural network learning dynamics for researchers, but it is incremental as it focuses on observational trends without proposing new methods or applications.
The study investigated learning dynamics in deep convolutional neural networks by measuring per-layer relative weight changes during training, revealing trends such as increased weight change in later layers compared to earlier ones across various architectures and computer vision tasks.
Understanding the per-layer learning dynamics of deep neural networks is of significant interest as it may provide insights into how neural networks learn and the potential for better training regimens. We investigate learning in Deep Convolutional Neural Networks (CNNs) by measuring the relative weight change of layers while training. Several interesting trends emerge in a variety of CNN architectures across various computer vision classification tasks, including the overall increase in relative weight change of later layers as compared to earlier ones.