The Neural Pruning Law Hypothesis
This work addresses the need for a unifying theoretical framework in neural network pruning, which could improve efficiency and understanding across AI applications, though it is incremental in building on existing pruning concepts.
The paper tackles the problem of poorly understood ad-hoc heuristics in neural network pruning by introducing Hyperflux, a method that models pruning as an interaction between weight flux and network pressure, leading to the discovery of a power-law relationship between minimum flux and density, which they hypothesize as the Neural Pruning Law applicable to various pruning methods.
Network pruning is used to reduce inference latency and power consumption in large neural networks. However, most current pruning methods rely on ad-hoc heuristics that are poorly understood. We introduce Hyperflux, a conceptually-grounded pruning method, and use it to study the pruning process. Hyperflux models this process as an interaction between weight flux, the gradient's response to the weight's removal, and network pressure, a global regularization driving weights towards pruning. We postulate properties that arise naturally from our framework and find that the relationship between minimum flux among weights and density follows a power-law equation. Furthermore, we hypothesize the power-law relationship to hold for any effective saliency metric and call this idea the Neural Pruning Law Hypothesis. We validate our hypothesis on several families of pruning methods (magnitude, gradients, $L_0$), providing a potentially unifying property for neural pruning.