An Operator Theoretic View on Pruning Deep Neural Networks
This work addresses a foundational theoretical gap in pruning methods for deep learning researchers, offering a unifying framework that is incremental but clarifies existing practices.
The paper tackles the lack of theoretical understanding behind why simple magnitude pruning works as well as complex methods in deep neural networks, and uses Koopman operator theory to unify magnitude and gradient-based pruning, providing insights into its early-training performance.
The discovery of sparse subnetworks that are able to perform as well as full models has found broad applied and theoretical interest. While many pruning methods have been developed to this end, the naïve approach of removing parameters based on their magnitude has been found to be as robust as more complex, state-of-the-art algorithms. The lack of theory behind magnitude pruning's success, especially pre-convergence, and its relation to other pruning methods, such as gradient based pruning, are outstanding open questions in the field that are in need of being addressed. We make use of recent advances in dynamical systems theory, namely Koopman operator theory, to define a new class of theoretically motivated pruning algorithms. We show that these algorithms can be equivalent to magnitude and gradient based pruning, unifying these seemingly disparate methods, and find that they can be used to shed light on magnitude pruning's performance during the early part of training.