Universality of Winning Tickets: A Renormalization Group Perspective
This provides a theoretical framework for studying transferable winning tickets, which is incremental but addresses a key gap in the Lottery Ticket Hypothesis literature.
The paper tackled the problem of understanding the universality of winning tickets across tasks and architectures by applying renormalization group theory, showing that iterative magnitude pruning acts as a renormalization group scheme and identifying common flow properties in ResNet-50 and BERT models, with evidence of near fixed points in BERT.
Foundational work on the Lottery Ticket Hypothesis has suggested an exciting corollary: winning tickets found in the context of one task can be transferred to similar tasks, possibly even across different architectures. This has generated broad interest, but methods to study this universality are lacking. We make use of renormalization group theory, a powerful tool from theoretical physics, to address this need. We find that iterative magnitude pruning, the principal algorithm used for discovering winning tickets, is a renormalization group scheme, and can be viewed as inducing a flow in parameter space. We demonstrate that ResNet-50 models with transferable winning tickets have flows with common properties, as would be expected from the theory. Similar observations are made for BERT models, with evidence that their flows are near fixed points. Additionally, we leverage our framework to study winning tickets transferred across ResNet architectures, observing that smaller models have flows with more uniform properties than larger models, complicating transfer between them.