A Generalized Lottery Ticket Hypothesis
This work provides a theoretical extension for neural network pruning, but it is incremental and preliminary, aimed at encouraging further research.
The authors generalized the lottery ticket hypothesis by relaxing sparsity to an arbitrary parameter basis, finding that original results hold in this broader setting and showing how structured pruning methods fit as instances of this generalization.
We introduce a generalization to the lottery ticket hypothesis in which the notion of "sparsity" is relaxed by choosing an arbitrary basis in the space of parameters. We present evidence that the original results reported for the canonical basis continue to hold in this broader setting. We describe how structured pruning methods, including pruning units or factorizing fully-connected layers into products of low-rank matrices, can be cast as particular instances of this "generalized" lottery ticket hypothesis. The investigations reported here are preliminary and are provided to encourage further research along this direction.