Signing the Supermask: Keep, Hide, Invert
This addresses the issue of high hardware requirements and interpretability for real-world applications, though it is incremental as it extends prior work on masking and the Lottery Ticket Hypothesis.
The authors tackled the problem of large, hard-to-interpret neural networks by introducing a method that selects and inverts initial weights without changing their absolute values, achieving up to 99% pruning while matching or exceeding baseline performance.
The exponential growth in numbers of parameters of neural networks over the past years has been accompanied by an increase in performance across several fields. However, due to their sheer size, the networks not only became difficult to interpret but also problematic to train and use in real-world applications, since hardware requirements increased accordingly. Tackling both issues, we present a novel approach that either drops a neural network's initial weights or inverts their respective sign. Put simply, a network is trained by weight selection and inversion without changing their absolute values. Our contribution extends previous work on masking by additionally sign-inverting the initial weights and follows the findings of the Lottery Ticket Hypothesis. Through this extension and adaptations of initialization methods, we achieve a pruning rate of up to 99%, while still matching or exceeding the performance of various baseline and previous models. Our approach has two main advantages. First, and most notable, signed Supermask models drastically simplify a model's structure, while still performing well on given tasks. Second, by reducing the neural network to its very foundation, we gain insights into which weights matter for performance. The code is available on GitHub.