RED : Looking for Redundancies for Data-Free Structured Compression of Deep Neural Networks
This addresses the computational cost issue in computer vision DNNs by enabling efficient pruning without data, which is incremental as it builds on existing structured pruning techniques.
The paper tackles the problem of accelerating deep neural networks without requiring data for retraining by introducing RED, a data-free structured pruning method that merges redundant neurons and uses uneven depthwise separation, achieving performance comparable to data-driven methods across various benchmarks.
Deep Neural Networks (DNNs) are ubiquitous in today's computer vision land-scape, despite involving considerable computational costs. The mainstream approaches for runtime acceleration consist in pruning connections (unstructured pruning) or, better, filters (structured pruning), both often requiring data to re-train the model. In this paper, we present RED, a data-free structured, unified approach to tackle structured pruning. First, we propose a novel adaptive hashing of the scalar DNN weight distribution densities to increase the number of identical neurons represented by their weight vectors. Second, we prune the network by merging redundant neurons based on their relative similarities, as defined by their distance. Third, we propose a novel uneven depthwise separation technique to further prune convolutional layers. We demonstrate through a large variety of benchmarks that RED largely outperforms other data-free pruning methods, often reaching performance similar to unconstrained, data-driven methods.