Linear Mode Connectivity and the Lottery Ticket Hypothesis
This work addresses the problem of understanding optimization stability and subnetwork trainability in neural networks for researchers in deep learning, providing incremental insights into the lottery ticket hypothesis.
The paper investigates whether neural networks converge to linearly connected minima under different SGD noise samples, finding that standard vision models become stable early in training, determining optimization outcomes to a linearly connected region. It applies this to iterative magnitude pruning from the lottery ticket hypothesis, showing subnetworks achieve full accuracy only when stable to SGD noise, occurring at initialization for small datasets like MNIST or early in training for large-scale models like ResNet-50 on ImageNet.
We study whether a neural network optimizes to the same, linearly connected minimum under different samples of SGD noise (e.g., random data order and augmentation). We find that standard vision models become stable to SGD noise in this way early in training. From then on, the outcome of optimization is determined to a linearly connected region. We use this technique to study iterative magnitude pruning (IMP), the procedure used by work on the lottery ticket hypothesis to identify subnetworks that could have trained in isolation to full accuracy. We find that these subnetworks only reach full accuracy when they are stable to SGD noise, which either occurs at initialization for small-scale settings (MNIST) or early in training for large-scale settings (ResNet-50 and Inception-v3 on ImageNet).