Sparse Weight Averaging with Multiple Particles for Iterative Magnitude Pruning
This work addresses the need for efficient sparse architectures to reduce inference time and memory usage in large neural networks, representing an incremental improvement over existing pruning methods.
The paper tackles the challenge of improving sparse neural networks by proposing Sparse Weight Averaging with Multiple Particles (SWAMP), a modification of Iterative Magnitude Pruning (IMP) that trains multiple sparse models concurrently and averages them, achieving performance comparable to an ensemble of two IMP solutions and consistently outperforming existing baselines across different sparsities in experiments.
Given the ever-increasing size of modern neural networks, the significance of sparse architectures has surged due to their accelerated inference speeds and minimal memory demands. When it comes to global pruning techniques, Iterative Magnitude Pruning (IMP) still stands as a state-of-the-art algorithm despite its simple nature, particularly in extremely sparse regimes. In light of the recent finding that the two successive matching IMP solutions are linearly connected without a loss barrier, we propose Sparse Weight Averaging with Multiple Particles (SWAMP), a straightforward modification of IMP that achieves performance comparable to an ensemble of two IMP solutions. For every iteration, we concurrently train multiple sparse models, referred to as particles, using different batch orders yet the same matching ticket, and then weight average such models to produce a single mask. We demonstrate that our method consistently outperforms existing baselines across different sparsities through extensive experiments on various data and neural network structures.