Geodesic Mode Connectivity
This addresses the challenge of understanding and navigating neural network loss landscapes for researchers in machine learning, but it appears incremental as it builds on existing mode connectivity concepts with a geometric twist.
The paper tackled the problem of connecting trained models via low-loss paths by reframing mode connectivity in the context of Information Geometry, hypothesizing that geodesics correspond to such paths, and demonstrated that their algorithm achieves mode connectivity.
Mode connectivity is a phenomenon where trained models are connected by a path of low loss. We reframe this in the context of Information Geometry, where neural networks are studied as spaces of parameterized distributions with curved geometry. We hypothesize that shortest paths in these spaces, known as geodesics, correspond to mode-connecting paths in the loss landscape. We propose an algorithm to approximate geodesics and demonstrate that they achieve mode connectivity.