Iterative Magnitude Pruning as a Renormalisation Group: A Study in The Context of The Lottery Ticket Hypothesis
This work addresses the problem of understanding and optimizing neural network pruning for researchers in machine learning, but it appears incremental as it builds on existing LTH and IMP concepts.
The paper investigates Iterative Magnitude Pruning in the context of the Lottery Ticket Hypothesis, exploring the universality of winning tickets across similar problems and linking IMP to Renormalisation Group theory for a more rigorous understanding.
This thesis delves into the intricate world of Deep Neural Networks (DNNs), focusing on the exciting concept of the Lottery Ticket Hypothesis (LTH). The LTH posits that within extensive DNNs, smaller, trainable subnetworks termed "winning tickets", can achieve performance comparable to the full model. A key process in LTH, Iterative Magnitude Pruning (IMP), incrementally eliminates minimal weights, emulating stepwise learning in DNNs. Once we identify these winning tickets, we further investigate their "universality". In other words, we check if a winning ticket that works well for one specific problem could also work well for other, similar problems. We also bridge the divide between the IMP and the Renormalisation Group (RG) theory in physics, promoting a more rigorous understanding of IMP.