Max-Affine Spline Insights Into Deep Network Pruning
This research provides a theoretical understanding of deep network pruning for practitioners who typically rely on random initialization, addressing the current empirical nature of the field.
This paper investigates the importance of pruning in deep networks (DNs) by analyzing the relationship between pruning highly overparameterized DNs and training small, cleverly initialized DNs. Using Continuous Piecewise Affine (CPA) DNs, the authors detect the early-bird ticket phenomenon, interpret existing pruning techniques, and develop a principled pruning strategy that performs on par with or outperforms state-of-the-art methods.
In this paper, we study the importance of pruning in Deep Networks (DNs) and the yin & yang relationship between (1) pruning highly overparametrized DNs that have been trained from random initialization and (2) training small DNs that have been "cleverly" initialized. As in most cases practitioners can only resort to random initialization, there is a strong need to develop a grounded understanding of DN pruning. Current literature remains largely empirical, lacking a theoretical understanding of how pruning affects DNs' decision boundary, how to interpret pruning, and how to design corresponding principled pruning techniques. To tackle those questions, we propose to employ recent advances in the theoretical analysis of Continuous Piecewise Affine (CPA) DNs. From this perspective, we will be able to detect the early-bird (EB) ticket phenomenon, provide interpretability into current pruning techniques, and develop a principled pruning strategy. In each step of our study, we conduct extensive experiments supporting our claims and results; while our main goal is to enhance the current understanding towards DN pruning instead of developing a new pruning method, our spline pruning criteria in terms of layerwise and global pruning is on par with or even outperforms state-of-the-art pruning methods.