Considering Layerwise Importance in the Lottery Ticket Hypothesis
This is an incremental improvement for neural network pruning research, addressing a specific limitation in the original Lottery Ticket Hypothesis.
The paper tackles the problem of losing layerwise context in the Lottery Ticket Hypothesis by generalizing it to use weight importance metrics instead of global weight magnitudes, finding that different metrics yield distinct high-performing sparse networks with little overlap, indicating lottery tickets are not unique.
The Lottery Ticket Hypothesis (LTH) showed that by iteratively training a model, removing connections with the lowest global weight magnitude and rewinding the remaining connections, sparse networks can be extracted. This global comparison removes context information between connections within a layer. Here we study means for recovering some of this layer distributional context and generalise the LTH to consider weight importance values rather than global weight magnitudes. We find that given a repeatable training procedure, applying different importance metrics leads to distinct performant lottery tickets with little overlapping connections. This strongly suggests that lottery tickets are not unique