Catalyst: a Novel Regularizer for Structured Pruning with Auxiliary Extension of Parameter Space
This work addresses a key bottleneck in model compression for deep learning practitioners by providing a more reliable and effective pruning method, though it is incremental as it builds on existing regularization approaches.
The paper tackles the problem of magnitude bias and instability in structured pruning of neural networks by introducing Catalyst, a novel regularizer that ensures fair pruning chances and robust decisions through wide-margin bifurcation, achieving superior results compared to state-of-the-art methods on various datasets and models.
Structured pruning aims to reduce the size and computational cost of deep neural networks by removing entire filters or channels. The traditional regularizers such as L1 or Group Lasso and its variants lead to magnitude-biased pruning decisions, such that the filters with small magnitudes are likely to be pruned. Also, they often entail pruning results with almost zero margin around pruning decision boundary, such that tiny perturbation in a filter magnitude can flip the pruning decision. In this paper, we identify the precise algebraic condition under which pruning operations preserve model performance, and use the condition to construct a novel regularizer defined in an extended parameter space via auxiliary catalyst variables. The proposed Catalyst regularization ensures fair pruning chance for each filters with theoretically provable zero bias to their magnitude and robust pruning behavior achieved by wide-margin bifurcation of magnitudes between the preserved and the pruned filters. The theoretical properties naturally lead to real-world effectiveness, as shown by empirical validations of Catalyst Pruning algorithm. Pruning results on various datasets and models are superior to state-of-the-art filter pruning methods, and at the same time confirm the predicted robust and fair pruning characteristics of Catalyst pruning.