Pruning artificial neural networks: a way to find well-generalizing, high-entropy sharp minima
This work addresses the lack of theoretical understanding in neural network pruning, offering insights that could improve model efficiency and generalization, though it is incremental in nature.
The paper tackles the problem of understanding why pruning strategies for deep networks are effective, finding that gradual pruning accesses narrow, well-generalizing minima and leads to features less correlated to specific classes, with potential benefits for transfer learning.
Recently, a race towards the simplification of deep networks has begun, showing that it is effectively possible to reduce the size of these models with minimal or no performance loss. However, there is a general lack in understanding why these pruning strategies are effective. In this work, we are going to compare and analyze pruned solutions with two different pruning approaches, one-shot and gradual, showing the higher effectiveness of the latter. In particular, we find that gradual pruning allows access to narrow, well-generalizing minima, which are typically ignored when using one-shot approaches. In this work we also propose PSP-entropy, a measure to understand how a given neuron correlates to some specific learned classes. Interestingly, we observe that the features extracted by iteratively-pruned models are less correlated to specific classes, potentially making these models a better fit in transfer learning approaches.