Randomly Initialized Subnetworks with Iterative Weight Recycling
This work addresses the problem of efficient neural network pruning for researchers and practitioners by offering a method that reduces storage and scaling costs, though it is incremental as it modifies existing algorithms.
The paper tackles the challenge of finding high-accuracy subnetworks in randomly initialized neural networks without requiring overparameterization, proposing Iterative Weight Recycling to improve sparsity and performance on smaller architectures and higher prune rates, with empirical results showing increased model sparsity through weight reuse.
The Multi-Prize Lottery Ticket Hypothesis posits that randomly initialized neural networks contain several subnetworks that achieve comparable accuracy to fully trained models of the same architecture. However, current methods require that the network is sufficiently overparameterized. In this work, we propose a modification to two state-of-the-art algorithms (Edge-Popup and Biprop) that finds high-accuracy subnetworks with no additional storage cost or scaling. The algorithm, Iterative Weight Recycling, identifies subsets of important weights within a randomly initialized network for intra-layer reuse. Empirically we show improvements on smaller network architectures and higher prune rates, finding that model sparsity can be increased through the "recycling" of existing weights. In addition to Iterative Weight Recycling, we complement the Multi-Prize Lottery Ticket Hypothesis with a reciprocal finding: high-accuracy, randomly initialized subnetwork's produce diverse masks, despite being generated with the same hyperparameter's and pruning strategy. We explore the landscapes of these masks, which show high variability.