Visualizing the Loss Landscape of Winning Lottery Tickets
This work addresses the computational bottleneck in visualizing loss landscapes for deep neural networks, providing insights into pruning methods, but it is incremental as it builds on existing techniques.
The paper tackled the problem of computationally expensive loss landscape visualization by developing a faster method, which was then used to analyze winning lottery tickets from iterative magnitude pruning, revealing contradictions with prior claims about correlations between loss landscape projections and model performance.
The underlying loss landscapes of deep neural networks have a great impact on their training, but they have mainly been studied theoretically due to computational constraints. This work vastly reduces the time required to compute such loss landscapes, and uses them to study winning lottery tickets found via iterative magnitude pruning. We also share results that contradict previously claimed correlations between certain loss landscape projection methods and model trainability and generalization error.