Exploring The Neural Burden In Pruned Models: An Insight Inspired By Neuroscience
This work addresses the problem of performance loss in model pruning for researchers and practitioners in AI compression, though it appears incremental as it builds on existing pruning-during-training techniques.
The paper investigates the performance decline in pruned neural networks by introducing the concept of Neural Burden, inspired by neuroscience, and proposes a method to mitigate this issue, showing its existence and potential through experiments.
Vision Transformer and its variants have been adopted in many visual tasks due to their powerful capabilities, which also bring significant challenges in computation and storage. Consequently, researchers have introduced various compression methods in recent years, among which the pruning techniques are widely used to remove a significant fraction of the network. Therefore, these methods can reduce significant percent of the FLOPs, but often lead to a decrease in model performance. To investigate the underlying causes, we focus on the pruning methods specifically belonging to the pruning-during-training category, then drew inspiration from neuroscience and propose a new concept for artificial neural network models named Neural Burden. We investigate its impact in the model pruning process, and subsequently explore a simple yet effective approach to mitigate the decline in model performance, which can be applied to any pruning-during-training technique. Extensive experiments indicate that the neural burden phenomenon indeed exists, and show the potential of our method. We hope that our findings can provide valuable insights for future research. Code will be made publicly available after this paper is published.