Sign-In to the Lottery: Reparameterizing Sparse Training From Scratch
This work addresses a bottleneck in efficient deep learning for researchers and practitioners by incrementally improving sparse training from scratch.
The paper tackles the performance gap in training sparse neural networks from scratch by proposing Sign-In, a method that reparameterizes parameters to induce sign flips, which improved PaI performance in experiments, though it did not fully close the gap with dense-to-sparse training.
The performance gap between training sparse neural networks from scratch (PaI) and dense-to-sparse training presents a major roadblock for efficient deep learning. According to the Lottery Ticket Hypothesis, PaI hinges on finding a problem specific parameter initialization. As we show, to this end, determining correct parameter signs is sufficient. Yet, they remain elusive to PaI. To address this issue, we propose Sign-In, which employs a dynamic reparameterization that provably induces sign flips. Such sign flips are complementary to the ones that dense-to-sparse training can accomplish, rendering Sign-In as an orthogonal method. While our experiments and theory suggest performance improvements of PaI, they also carve out the main open challenge to close the gap between PaI and dense-to-sparse training.