Structural Pruning in Deep Neural Networks: A Small-World Approach
This addresses memory and hardware costs for deploying deep learning models, but it is incremental as it builds on existing pruning methods by incorporating network structure insights.
The paper tackles the problem of over-parameterization in deep neural networks by proposing a structural pruning scheme based on the Small-World model, which reduces model size and interconnection before training, achieving parameter reductions to 2.3% for LeNet-5 on MNIST and 9.02% for VGG-16 on CIFAR-10.
Deep Neural Networks (DNNs) are usually over-parameterized, causing excessive memory and interconnection cost on the hardware platform. Existing pruning approaches remove secondary parameters at the end of training to reduce the model size; but without exploiting the intrinsic network property, they still require the full interconnection to prepare the network. Inspired by the observation that brain networks follow the Small-World model, we propose a novel structural pruning scheme, which includes (1) hierarchically trimming the network into a Small-World model before training, (2) training the network for a given dataset, and (3) optimizing the network for accuracy. The new scheme effectively reduces both the model size and the interconnection needed before training, achieving a locally clustered and globally sparse model. We demonstrate our approach on LeNet-5 for MNIST and VGG-16 for CIFAR-10, decreasing the number of parameters to 2.3% and 9.02% of the baseline model, respectively.