Statistical Mechanical Analysis of Neural Network Pruning
This provides theoretical guarantees for pruning techniques, benefiting researchers and practitioners in deep learning by explaining empirical observations, though it is incremental as it builds on existing frameworks.
The authors tackled the lack of theoretical understanding of neural network pruning methods by analyzing them under a statistical mechanics framework, deriving generalization error bounds that theoretically justify empirical findings, such as DPP node pruning's superiority and sparse networks generalizing better than dense ones.
Deep learning architectures with a huge number of parameters are often compressed using pruning techniques to ensure computational efficiency of inference during deployment. Despite multitude of empirical advances, there is a lack of theoretical understanding of the effectiveness of different pruning methods. We inspect different pruning techniques under the statistical mechanics formulation of a teacher-student framework and derive their generalization error (GE) bounds. It has been shown that Determinantal Point Process (DPP) based node pruning method is notably superior to competing approaches when tested on real datasets. Using GE bounds in the aforementioned setup we provide theoretical guarantees for their empirical observations. Another consistent finding in literature is that sparse neural networks (edge pruned) generalize better than dense neural networks (node pruned) for a fixed number of parameters. We use our theoretical setup to prove this finding and show that even the baseline random edge pruning method performs better than the DPP node pruning method. We also validate this empirically on real datasets.