Dynamical Isometry: The Missing Ingredient for Neural Network Pruning
This provides a theoretical explanation for a key phenomenon in neural network pruning, benefiting researchers and practitioners in model compression.
The paper tackles the problem of why a larger finetuning learning rate improves neural network pruning performance, explaining it through dynamical isometry by viewing pruning as initialization for finetuning and showing that inherited weights are poor initializations, which also clarifies the value of pruning.
Several recent works [40, 24] observed an interesting phenomenon in neural network pruning: A larger finetuning learning rate can improve the final performance significantly. Unfortunately, the reason behind it remains elusive up to date. This paper is meant to explain it through the lens of dynamical isometry [42]. Specifically, we examine neural network pruning from an unusual perspective: pruning as initialization for finetuning, and ask whether the inherited weights serve as a good initialization for the finetuning? The insights from dynamical isometry suggest a negative answer. Despite its critical role, this issue has not been well-recognized by the community so far. In this paper, we will show the understanding of this problem is very important -- on top of explaining the aforementioned mystery about the larger finetuning rate, it also unveils the mystery about the value of pruning [5, 30]. Besides a clearer theoretical understanding of pruning, resolving the problem can also bring us considerable performance benefits in practice.