Plant 'n' Seek: Can You Find the Winning Ticket?
This work addresses the challenge of understanding and improving pruning algorithms for deep learning efficiency, but it is incremental as it builds on existing lottery ticket hypothesis research by providing a new evaluation framework.
The authors tackled the problem of evaluating pruning algorithms for finding sparse lottery tickets in neural networks by developing a framework to plant and hide winning tickets with known properties. They found that current pruning methods show similar trends as in imaging studies but have limitations in identifying extremely sparse tickets, suggesting these limitations are algorithmic rather than fundamental.
The lottery ticket hypothesis has sparked the rapid development of pruning algorithms that aim to reduce the computational costs associated with deep learning during training and model deployment. Currently, such algorithms are primarily evaluated on imaging data, for which we lack ground truth information and thus the understanding of how sparse lottery tickets could be. To fill this gap, we develop a framework that allows us to plant and hide winning tickets with desirable properties in randomly initialized neural networks. To analyze the ability of state-of-the-art pruning to identify tickets of extreme sparsity, we design and hide such tickets solving four challenging tasks. In extensive experiments, we observe similar trends as in imaging studies, indicating that our framework can provide transferable insights into realistic problems. Additionally, we can now see beyond such relative trends and highlight limitations of current pruning methods. Based on our results, we conclude that the current limitations in ticket sparsity are likely of algorithmic rather than fundamental nature. We anticipate that comparisons to planted tickets will facilitate future developments of efficient pruning algorithms.