Data-dependent Pruning to find the Winning Lottery Ticket
This work addresses the problem of efficient neural network pruning for researchers and practitioners, but it is incremental as it builds on prior methods like SNIP.
The paper investigates methods to find subnetworks that match full network performance, showing that adding a data-dependent gradient component to pruning criteria consistently improves existing algorithms.
The Lottery Ticket Hypothesis postulates that a freshly initialized neural network contains a small subnetwork that can be trained in isolation to achieve similar performance as the full network. Our paper examines several alternatives to search for such subnetworks. We conclude that incorporating a data dependent component into the pruning criterion in the form of the gradient of the training loss -- as done in the SNIP method -- consistently improves the performance of existing pruning algorithms.