Optimal Brain Connection: Towards Efficient Structural Pruning
This work addresses the challenge of efficiently compressing neural networks for deployment, but it appears incremental as it builds on existing pruning methods by focusing on parameter interconnections.
The paper tackles the problem of structural pruning in neural networks by proposing a framework that addresses neglected interconnections among parameters, resulting in a Jacobian Criterion that outperforms existing metrics and an Equivalent Pruning mechanism that reduces performance degradation after fine-tuning.
Structural pruning has been widely studied for its effectiveness in compressing neural networks. However, existing methods often neglect the interconnections among parameters. To address this limitation, this paper proposes a structural pruning framework termed Optimal Brain Connection. First, we introduce the Jacobian Criterion, a first-order metric for evaluating the saliency of structural parameters. Unlike existing first-order methods that assess parameters in isolation, our criterion explicitly captures both intra-component interactions and inter-layer dependencies. Second, we propose the Equivalent Pruning mechanism, which utilizes autoencoders to retain the contributions of all original connection--including pruned ones--during fine-tuning. Experimental results demonstrate that the Jacobian Criterion outperforms several popular metrics in preserving model performance, while the Equivalent Pruning mechanism effectively mitigates performance degradation after fine-tuning. Code: https://github.com/ShaowuChen/Optimal_Brain_Connection