Towards Scalable Lottery Ticket Networks using Genetic Algorithms
This work addresses the need for scalable and efficient deep learning models by providing a method to skip expensive training, which is incremental as it builds on the Strong Lottery Ticket Hypothesis with a new algorithmic approach.
The paper tackles the problem of finding efficient subnetworks within randomly initialized neural networks without training, using genetic algorithms to identify strong lottery tickets. It achieves better accuracies and sparsity levels than the state-of-the-art on binary and multi-class classification tasks.
Building modern deep learning systems that are not just effective but also efficient requires rethinking established paradigms for model training and neural architecture design. Instead of adapting highly overparameterized networks and subsequently applying model compression techniques to reduce resource consumption, a new class of high-performing networks skips the need for expensive parameter updates, while requiring only a fraction of parameters, making them highly scalable. The Strong Lottery Ticket Hypothesis posits that within randomly initialized, sufficiently overparameterized neural networks, there exist subnetworks that can match the accuracy of the trained original model-without any training. This work explores the usage of genetic algorithms for identifying these strong lottery ticket subnetworks. We find that for instances of binary and multi-class classification tasks, our approach achieves better accuracies and sparsity levels than the current state-of-the-art without requiring any gradient information. In addition, we provide justification for the need for appropriate evaluation metrics when scaling to more complex network architectures and learning tasks.