ONG: One-Shot NMF-based Gradient Masking for Efficient Model Sparsification
This addresses deployment challenges for deep learning models by providing a more efficient pruning method, though it appears incremental as it builds on existing sparsification strategies.
The paper tackled the problem of inefficient and complex pruning techniques for deep neural networks by introducing ONG, a one-shot sparsification method using Non-negative Matrix Factorization and gradient masking, which achieved comparable or superior performance on CIFAR-10 and CIFAR-100 with ResNet models at various sparsity levels.
Deep Neural Networks (DNNs) have achieved remarkable success but their large size poses deployment challenges. While various pruning techniques exist, many involve complex iterative processes, specialized criteria, or struggle to maintain sparsity effectively during training. We introduce ONG (One-shot NMF-based Gradient Masking), a novel sparsification strategy that identifies salient weight structures using Non-negative Matrix Factorization (NMF) for one-shot pruning at the outset of training. Subsequently, ONG employs a precise gradient masking mechanism to ensure that only unpruned weights are updated, strictly preserving the target sparsity throughout the training phase. We integrate ONG into the BIMP comparative framework and evaluate it on CIFAR-10 and CIFAR-100 with ResNet56, ResNet34, and ResNet18 against established stable sparsification methods. Our experiments demonstrate ONG's ability to achieve comparable or superior performance at various sparsity levels while maintaining structural integrity post-pruning and offering a clear mechanism for targeting desired sparsities.