Deconstructing the Structure of Sparse Neural Networks
This work provides insights into the underlying mechanisms of sparse neural networks, which is important for researchers and practitioners aiming to develop more efficient and robust deep learning models.
This paper investigates the structural properties of sparse neural networks, revealing that network accuracy can be largely attributed to structure alone, even with different random weight initializations. It also finds that the structure for a dynamic sparsity algorithm is mostly determined after just one epoch, enabling a more efficient training approach.
Although sparse neural networks have been studied extensively, the focus has been primarily on accuracy. In this work, we focus instead on network structure, and analyze three popular algorithms. We first measure performance when structure persists and weights are reset to a different random initialization, thereby extending experiments in Deconstructing Lottery Tickets (Zhou et al., 2019). This experiment reveals that accuracy can be derived from structure alone. Second, to measure structural robustness we investigate the sensitivity of sparse neural networks to further pruning after training, finding a stark contrast between algorithms. Finally, for a recent dynamic sparsity algorithm we investigate how early in training the structure emerges. We find that even after one epoch the structure is mostly determined, allowing us to propose a more efficient algorithm which does not require dense gradients throughout training. In looking back at algorithms for sparse neural networks and analyzing their performance from a different lens, we uncover several interesting properties and promising directions for future research.