Deep Neural Compression Via Concurrent Pruning and Self-Distillation
This work addresses the need for efficient and high-performing language models, offering an incremental improvement by integrating distillation into pruning without a separate student network.
The paper tackles the problem of reducing neural network parameters via pruning while maintaining performance, by proposing a self-distillation-based pruning strategy that maximizes representational similarity between pruned and unpruned versions. Results show that self-distilled pruned models outperform smaller Transformers with equal parameters and are competitive against larger distilled networks on GLUE and XGLUE benchmarks.
Pruning aims to reduce the number of parameters while maintaining performance close to the original network. This work proposes a novel \emph{self-distillation} based pruning strategy, whereby the representational similarity between the pruned and unpruned versions of the same network is maximized. Unlike previous approaches that treat distillation and pruning separately, we use distillation to inform the pruning criteria, without requiring a separate student network as in knowledge distillation. We show that the proposed {\em cross-correlation objective for self-distilled pruning} implicitly encourages sparse solutions, naturally complementing magnitude-based pruning criteria. Experiments on the GLUE and XGLUE benchmarks show that self-distilled pruning increases mono- and cross-lingual language model performance. Self-distilled pruned models also outperform smaller Transformers with an equal number of parameters and are competitive against (6 times) larger distilled networks. We also observe that self-distillation (1) maximizes class separability, (2) increases the signal-to-noise ratio, and (3) converges faster after pruning steps, providing further insights into why self-distilled pruning improves generalization.