How many winning tickets are there in one DNN?
This work addresses a foundational problem in deep learning theory by revealing the multiplicity of winning tickets, which could impact pruning and optimization strategies for researchers and practitioners.
The paper challenges the lottery ticket hypothesis by demonstrating that multiple distinct winning sub-networks exist within a single DNN, even with fixed initial weights, and these sub-networks show no significant overlap or correlation beyond chance, indicating a distribution over capable sub-networks rather than a single one.
The recent lottery ticket hypothesis proposes that there is one sub-network that matches the accuracy of the original network when trained in isolation. We show that instead each network contains several winning tickets, even if the initial weights are fixed. The resulting winning sub-networks are not instances of the same network under weight space symmetry, and show no overlap or correlation significantly larger than expected by chance. If randomness during training is decreased, overlaps higher than chance occur, even if the networks are trained on different tasks. We conclude that there is rather a distribution over capable sub-networks, as opposed to a single winning ticket.