Disentangling Linear Mode-Connectivity
This work addresses a foundational problem in machine learning for researchers studying neural network optimization, but it is incremental as it builds on existing empirical observations without introducing new methods or broad SOTA results.
The paper tackled the lack of theoretical understanding and systematic study of linear mode-connectivity (LMC) in neural network loss landscapes by exploring how LMC is affected by architecture, training strategy, and dataset, aiming to guide future theoretical works.
Linear mode-connectivity (LMC) (or lack thereof) is one of the intriguing characteristics of neural network loss landscapes. While empirically well established, it unfortunately still lacks a proper theoretical understanding. Even worse, although empirical data points are abound, a systematic study of when networks exhibit LMC is largely missing in the literature. In this work we aim to close this gap. We explore how LMC is affected by three factors: (1) architecture (sparsity, weight-sharing), (2) training strategy (optimization setup) as well as (3) the underlying dataset. We place particular emphasis on minimal but non-trivial settings, removing as much unnecessary complexity as possible. We believe that our insights can guide future theoretical works on uncovering the inner workings of LMC.