Signal Collapse in One-Shot Pruning: When Sparse Models Fail to Distinguish Neural Representations
This addresses the challenge of deploying efficient models on resource-constrained hardware by improving pruning accuracy, though it is incremental as it builds on existing pruning methods.
The paper tackles the problem of accuracy loss in neural network pruning by identifying signal collapse as the root cause, and proposes REFLOW to mitigate it, enabling magnitude pruning to restore ResNeXt101 accuracy from under 4.1% to 78.9% on ImageNet with 20% weights retained.
Neural network pruning is essential for reducing model complexity to enable deployment on resource constrained hardware. While performance loss of pruned networks is often attributed to the removal of critical parameters, we identify signal collapse a reduction in activation variance across layers as the root cause. Existing one shot pruning methods focus on weight selection strategies and rely on computationally expensive second order approximations. In contrast, we demonstrate that mitigating signal collapse, rather than optimizing weight selection, is key to improving accuracy of pruned networks. We propose REFLOW that addresses signal collapse without updating trainable weights, revealing high quality sparse sub networks within the original parameter space. REFLOW enables magnitude pruning to achieve state of the art performance, restoring ResNeXt101 accuracy from under 4.1% to 78.9% on ImageNet with only 20% of the weights retained, surpassing state of the art approaches.