Using Mode Connectivity for Loss Landscape Analysis
This work provides insights into loss landscape analysis for machine learning researchers, but it is incremental as it extends an existing framework without major breakthroughs.
The paper investigates the robustness of mode connectivity in connecting minima of neural networks under extreme training and initialization differences, finding it resilient, and applies this framework to analyze SGDR, showing that claims about it converging to and escaping multiple local minima are not empirically supported.
Mode connectivity is a recently introduced frame- work that empirically establishes the connected- ness of minima by finding a high accuracy curve between two independently trained models. To investigate the limits of this setup, we examine the efficacy of this technique in extreme cases where the input models are trained or initialized differently. We find that the procedure is resilient to such changes. Given this finding, we propose using the framework for analyzing loss surfaces and training trajectories more generally, and in this direction, study SGD with cosine annealing and restarts (SGDR). We report that while SGDR moves over barriers in its trajectory, propositions claiming that it converges to and escapes from multiple local minima are not substantiated by our empirical results.