Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptions
This addresses the need for more practical and theoretically guaranteed neural network compression for real-world applications, though it is incremental as it builds on existing coreset methods.
The paper tackles the problem of pruning deep neural networks by proposing a robust coreset-based framework that works under mild assumptions on model weights and no assumptions on training data, achieving a 62% compression rate on ResNet50 on ImageNet with only a 1.09% drop in accuracy.
Pruning is one of the predominant approaches for compressing deep neural networks (DNNs). Lately, coresets (provable data summarizations) were leveraged for pruning DNNs, adding the advantage of theoretical guarantees on the trade-off between the compression rate and the approximation error. However, coresets in this domain were either data-dependent or generated under restrictive assumptions on both the model's weights and inputs. In real-world scenarios, such assumptions are rarely satisfied, limiting the applicability of coresets. To this end, we suggest a novel and robust framework for computing such coresets under mild assumptions on the model's weights and without any assumption on the training data. The idea is to compute the importance of each neuron in each layer with respect to the output of the following layer. This is achieved by a combination of Löwner ellipsoid and Caratheodory theorem. Our method is simultaneously data-independent, applicable to various networks and datasets (due to the simplified assumptions), and theoretically supported. Experimental results show that our method outperforms existing coreset based neural pruning approaches across a wide range of networks and datasets. For example, our method achieved a $62\%$ compression rate on ResNet50 on ImageNet with $1.09\%$ drop in accuracy.