REPAIR: REnormalizing Permuted Activations for Interpolation Repair
This addresses the issue of loss barriers in neural network interpolation for researchers, but it is incremental as it builds on prior work on permutation invariance.
The paper tackled the problem of poor performance in linearly interpolated neural networks due to variance collapse in activations, and proposed REPAIR to rescale preactivations, achieving 60%-100% relative barrier reduction, including 74% for ResNet50 on ImageNet and 90% for ResNet18 on CIFAR10.
In this paper we look into the conjecture of Entezari et al. (2021) which states that if the permutation invariance of neural networks is taken into account, then there is likely no loss barrier to the linear interpolation between SGD solutions. First, we observe that neuron alignment methods alone are insufficient to establish low-barrier linear connectivity between SGD solutions due to a phenomenon we call variance collapse: interpolated deep networks suffer a collapse in the variance of their activations, causing poor performance. Next, we propose REPAIR (REnormalizing Permuted Activations for Interpolation Repair) which mitigates variance collapse by rescaling the preactivations of such interpolated networks. We explore the interaction between our method and the choice of normalization layer, network width, and depth, and demonstrate that using REPAIR on top of neuron alignment methods leads to 60%-100% relative barrier reduction across a wide variety of architecture families and tasks. In particular, we report a 74% barrier reduction for ResNet50 on ImageNet and 90% barrier reduction for ResNet18 on CIFAR10.