UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformer
This work addresses pruning challenges for efficient models like ConvNeXt and vision transformers, offering a novel method that is incremental but improves specific performance gains.
The paper tackled the problem of pruning efficient CNN models and vision transformers by introducing a unified progressive depth pruner that overcomes limitations of traditional methods, resulting in pruned models that outperform existing depth pruning methods and achieve state-of-the-art performance on vision transformers.
Traditional channel-wise pruning methods by reducing network channels struggle to effectively prune efficient CNN models with depth-wise convolutional layers and certain efficient modules, such as popular inverted residual blocks. Prior depth pruning methods by reducing network depths are not suitable for pruning some efficient models due to the existence of some normalization layers. Moreover, finetuning subnet by directly removing activation layers would corrupt the original model weights, hindering the pruned model from achieving high performance. To address these issues, we propose a novel depth pruning method for efficient models. Our approach proposes a novel block pruning strategy and progressive training method for the subnet. Additionally, we extend our pruning method to vision transformer models. Experimental results demonstrate that our method consistently outperforms existing depth pruning methods across various pruning configurations. We obtained three pruned ConvNeXtV1 models with our method applying on ConvNeXtV1, which surpass most SOTA efficient models with comparable inference performance. Our method also achieves state-of-the-art pruning performance on the vision transformer model.