Partially Frozen Random Networks Contain Compact Strong Lottery Tickets
This work addresses the memory efficiency of SLTs for neural network compression, offering an incremental improvement over prior methods.
The paper tackles the problem of compressing strong lottery tickets (SLTs) in neural networks without restricting sparsity, by freezing a random subset of initial weights to reduce model size. The result shows that freezing 70% of a ResNet on ImageNet provides 3.3x compression compared to dense SLTs, raises accuracy by up to 14.12 points compared to randomly pruned SLTs, and offers a better accuracy-model size trade-off.
Randomly initialized dense networks contain subnetworks that achieve high accuracy without weight learning--strong lottery tickets (SLTs). Recently, Gadhikar et al. (2023) demonstrated that SLTs could also be found within a randomly pruned source network. This phenomenon can be exploited to further compress the small memory size required by SLTs. However, their method is limited to SLTs that are even sparser than the source, leading to worse accuracy due to unintentionally high sparsity. This paper proposes a method for reducing the SLT memory size without restricting the sparsity of the SLTs that can be found. A random subset of the initial weights is frozen by either permanently pruning them or locking them as a fixed part of the SLT, resulting in a smaller model size. Experimental results show that Edge-Popup (Ramanujan et al., 2020; Sreenivasan et al., 2022) finds SLTs with better accuracy-to-model size trade-off within frozen networks than within dense or randomly pruned source networks. In particular, freezing $70\%$ of a ResNet on ImageNet provides $3.3 \times$ compression compared to the SLT found within a dense counterpart, raises accuracy by up to $14.12$ points compared to the SLT found within a randomly pruned counterpart, and offers a better accuracy-model size trade-off than both.